geographika / mappyfile Goto Github PK

The following CLASS definition:

    CLASS
      TEXT ([area])
    END

Drops the brackets following parsing:

    CLASS
      TEXT [area]
    END

Same for:

    CLASS
      TEXT ("[area]")
    END

See test_class_expression1 and test_class_expression2 in tests/test_snippets.py

As per http://mapserver.org/mapfile/include.html "File location can be given as a full path to the file, or (in MapServer >= 4.10.1) as a path relative to the mapfile".

Currently includes are relative to their parent file. This should be changed to the parent Mapfile (when not passing a string in directly) to be consistent with MapServer.

I encountered parsing issue with this project (which is really nice BTW). After doing some digging, it was caused by:

  PROJECTION "init=epsg:2056"
  END

I was able to correct the problem by replacing this block by:

  PROJECTION
    "init=epsg:2056"
  END

But the map file is parsed correctly by shp2img and by mapserv. I did some tests with CLASS NAME "Graticule", LABEL COLOR 0 0 0, GRID LABELFORMAT '%.0f' and PROJECTION "init=epsg:4326" END and the map files is parsed without any problem.

So I guess the grammar definition should use _NL* instead of _NL+. What do you think? I can do a PR for this if you want.

This is (a part of) code I use to open a template of a MapFile and set some values:

import mappyfile

d = mappyfile.open('test.map')


d["NAME"] = "testpg1"
d["PROJECTION"] = "init=epsg:4326" 

This is (a part of) the resulting MapFile:

MAP
    NAME "testpg1"
    PROJECTION
        "i"
        "n"
        "i"
        "t"
        "="
        "e"
        "p"
        "s"
        "g"
        ":"
        "4"
        "3"
        "2"
        "6"
    END

For some reason, the value I set for PROJECTION appears in the MapFile like this - each character on a new line. All the other parameters I set (like NAME) are ok. I have tried with several different MapFiles and it behaved the same way.

I am new to MapServer and mappyfile so there might be something very obvious that I am overlooking.

Regular expression case flags such as i become separated from their string e.g.

    LAYER
        NAME 'filters_test002'
        FILTER 'aitkin'i
    END

When parsed and pretty printed becomes:

    LAYER
        NAME 'filters_test002'
        FILTER 'aitkin' i
    END

More details on MapServer expressions at http://www.mapserver.org/mapfile/expressions.html

See test_filter in tests/test_snippets.py

This may be better handled in the pprint module than the parser.

from mappyfile.parser import Parser                                             
from mappyfile.pprint import PrettyPrinter                                      
from mappyfile.transformer import MapfileToDict                                 
from mappyfile.validator import Validator                                       
                                                                                
                                                                                
                                                                                
def output(s, include_position=True, schema_name="map"):                        
    """                                                                         
    Parse, transform, validate, and pretty print                                
    the result                                                                  
    """                                                                         
    p = Parser()                                                                
    m = MapfileToDict(include_position=include_position)                        
    ast = p.parse(s)                                                            
    d = m.transform(ast)                                                        
    v = Validator()                                                             
    errors = v.validate(d, schema_name=schema_name)                             
    pp = PrettyPrinter(indent=0, newlinechar=" ", quote="'")                    
    s = pp.pprint(d)                                                            
    print s                                                                     
    assert(len(errors) == 0)                                                    
    return s                                                                    
                                                                                
                                                                                
s = """                                                                         
STYLE                                                                           
  SIZE 0                                                                        
  GEOMTRANSFORM "end"                                                           
END                                                                             
"""                                                                             
                                                                                
output(s, schema_name="style") 

Result:

STYLE SIZE 0 GEOMTRANSFORM END END

As a devops person, I would like to install mappyfile via apt-get so that installations can be reproducible.

Many projects require Debian packaging setup for Ubuntu/Debian environments.

Mappyfile doesn't seem to like using || in EXPRESSIONs:

EXPRESSION ("[style_class]" = "10" || "[style_class]" = "20")

results in:

Traceback (most recent call last):
File "create-mc-config.py", line 135, in
mapfile = mappyfile.load('../mapserver/mapfiles/' + mapfileName)
File "/usr/local/lib/python2.7/site-packages/mappyfile/utils.py", line 9, in load
ast = p.parse_file(fn)
File "/usr/local/lib/python2.7/site-packages/mappyfile/parser.py", line 57, in parse_file
return self.parse(text)
File "/usr/local/lib/python2.7/site-packages/mappyfile/parser.py", line 73, in parse
return self.g.parse(text)
File "/Users//Library/Python/2.7/lib/python/site-packages/plyplus/plyplus.py", line 575, in parse
return self._grammar.parse(text)
File "/Users//Library/Python/2.7/lib/python/site-packages/plyplus/plyplus.py", line 700, in parse
tree = self.engine.parse(text)
File "/Users//Library/Python/2.7/lib/python/site-packages/plyplus/engine_pearley.py", line 92, in parse
tok = self.lexer.token()
File "/Users//Library/Python/2.7/lib/python/site-packages/plyplus/plyplus.py", line 450, in token
t = self.lexer.token()
File "/Users//Library/Python/2.7/lib/python/site-packages/ply/lex.py", line 386, in token
newtok = self.lexerrorf(tok)
File "/Users//Library/Python/2.7/lib/python/site-packages/plyplus/engine_ply.py", line 22, in t_error
raise TokenizeError("Illegal character in input: '%s', line: %s, %s" % (t.value[:32], t.lineno, t.type))
plyplus.common.TokenizeError: Illegal character in input: '|| "[style_class]" = "20")
ST', line: 1038, error
[Finished in 0.4s with exit code 1]

Using OR like so:

EXPRESSION ("[style_class]" = "10" OR "[style_class]" = "20")

works fine.

Hello,
I want to convert my mapfiles in json to queries them with jq...

I do a simple example, just a load, with a mapserver.map in the same directory (full access on it),
root@d40f7c90b3d7:/shells# head ./mapserver.map

MAP
    CONFIG  "MS_ERRORFILE" "/var/log/mapserver/error.log"
    NAME "test"
    EXTENT  -357823.2365 6037008.6939 1313632.3628 7230727.3772
    PROJECTION
        "init=epsg:2154"
    END
    DEBUG 4

root@d40f7c90b3d7:/shells# cat displaymapfile.py

import mappyfile
mapfile = mappyfile.load("./mapserver.map")

root@d40f7c90b3d7:/shells# python displaymapfile.py

Traceback (most recent call last):
  File "displaymapfile.py", line 2, in <module>
    mapfile = mappyfile.load("./mapserver.map")
  File "/usr/local/lib/python2.7/site-packages/mappyfile/utils.py", line 59, in load
    ast = p.load(fp)
  File "/usr/local/lib/python2.7/site-packages/mappyfile/parser.py", line 162, in load
    text = fp.read()
AttributeError: 'str' object has no attribute 'read'

I try with python 2.7 and python 3.5 (docker images) and install mappyfile with pip

Where's my mistake ?

BR,

    CLASS
      TEXT ("Area is: " + tostring([area],"%.2f"))
    END

Is not fully transformed and becomes:

  CLASS 
    TEXT Tree(add, ['"Area is: "', u'(tostring([area],"%.2f"))']) 
  END

See test_complex_class_expression in tests/test_snippets.py

See https://mapserver.org/mapfile/style.html. Currently hexadecimal colors with a translucence code throw a validation error.

RGBA value (adding translucence): “#rrggbbaa”. To specify a semi-translucent magenta, the following is used:

COLOR "#FF00FFCC"

During packaging there's an issue with running test

[   20s] + /usr/bin/python3 setup.py test
[   20s] running test
[   20s] running egg_info
[   20s] writing mappyfile.egg-info/PKG-INFO
[   20s] writing dependency_links to mappyfile.egg-info/dependency_links.txt
[   20s] writing requirements to mappyfile.egg-info/requires.txt
[   20s] writing top-level names to mappyfile.egg-info/top_level.txt
[   20s] reading manifest file 'mappyfile.egg-info/SOURCES.txt'
[   20s] reading manifest template 'MANIFEST.in'
[   20s] writing manifest file 'mappyfile.egg-info/SOURCES.txt'
[   20s] running build_ext
[   20s] error: [Errno 2] No such file or directory: 'C:\\VirtualEnvs\\mapserverdev\\Scripts\\activate_this.py'
[   20s] error: Bad exit status from /var/tmp/rpm-tmp.5rKNSp (%check)

For evident reason there's no C:\ drive under linux hosts ;-)
Also downloading something is strictly forbidden during build (happen in one shot vm with no external network access).
As packager I can grab myself before build a tar.gz containing alll .map from mapserver.org, and inject it at the right place for running the tests.

The function mappyfile.findall() takes the specified value and adds single and double quotes as possible values to search for:

But the result of the parsed mapfile contains the values (e.g. group or layer names) without any quotes and mappyfile.findall() always returns an empty generator object.

In my case, I could fix it by passing the specified value directly:

return (item for item in lst if item[key.lower()] == value)

Is there something wrong with my parsed content or needs this function to be fixed? mappyfile.find() works fine.

Parsing the following:

    LABEL
      COLOR  0 0 0
      FONT Vera
      TYPE truetype
      SIZE 8
      POSITION AUTO
      PARTIALS FALSE
      OUTLINECOLOR 255 255 255			
    END 

doesn't output AUTO - the AUTO is dropped.

AST looks as follows:

[Tree(attr_name, [Token(NAME, 'POSITION')]), Tree(bare_string, [])])

Likely due to following in grammar file:

bare_string: NAME | "SYMBOL"i | "AUTO"i | "GRID"i | "CLASS"i

See test_auto_projection

    MAP
        PROJECTION
            AUTO
        END
    END

Could be handled with the following grammar change:

projection: "PROJECTION"i _NL* ((string _NL*)+|bare_string _NL+) _END

However this would allow any bare_string, so may be better to add a new AUTO token just for this case.

layer = """                                                                     
LAYER                                                                           
    CONNECTION "user=user password=pass host=host dbname=mydb port=5432 options='-c client_encoding=utf-8'"
    CONNECTIONTYPE POSTGIS                                                      
    DATA "the_geom from  (SELECT *,mod(exposurecurve_level_db::integer, 5) as bgdi_modulo_5 FROM bazl.laermbelastungskataster_zivilflugplaetze ) as data using unique bgdi_id using srid=21781"
    NAME "ch.bazl.laermbelastungskataster-zivilflugplaetze_militaer-gesamt"        
    STATUS OFF                                                                  
    TEMPLATE "ttt"                                                              
    TYPE LINE                                                                   
    UNITS METERS                                                                
    LABELITEM exposurecurve_level_db                                            
    LABELMAXSCALEDENOM 20000                                                    
    FILTER "exposuregroup_exposuretype like 'OverallTrafficMilitary_Lr'"        
    METADATA                                                                    
        "ows_title"             "ch.bazl.laermbelastungskataster-zivilflugplaetze_militaer-gesamt"
        "ows_featureid"         "bgdi_id"                                       
        "wms_srs" "EPSG:21781 EPSG:2056 EPSG:4326 EPSG:3857 EPSG:3034 EPSG:3035 EPSG:4258 EPSG:31287 EPSG:25832 EPSG:25833 EPSG:31467 EPSG:32632 EPSG:32633"
        "ows_include_items"     "all"                                           
        "ows_exclude_items"     "bgdi_created_by,bgdi_modified_by,bgdi_created,bgdi_modified,bgdi_modulo_5"
    END                                                                         
    CLASS                                                                       
       NAME "bazl_overalltrafficmilitary_lr_5db"                                
       EXPRESSION ( [bgdi_modulo_5] eq 0 )                                      
       STYLE                                                                    
          COLOR 120 145 60                                                      
          WIDTH 2                                                               
       END                                                                      
       LABEL                                                                    
          COLOR 120 145 60                                                      
          OUTLINECOLOR 255 255 255                                              
          FONT "arialbd"                                                        
          TYPE truetype                                                         
          SIZE 10                                                               
          ANGLE FOLLOW                                                          
          MAXOVERLAPANGLE 40                                                    
          FORCE TRUE                                                            
       END                                                                      
    END                                                                         
                                                                                
    CLASS                                                                       
       NAME "bazl_overalltrafficmilitary_lr_1db"                                
       EXPRESSION ( [bgdi_modulo_5] != 0 )                                      
       STYLE                                                                    
          COLOR 120 145 60                                                      
          WIDTH 1                                                               
       END                                                                      
    END                                                                         
END                                                                                                                                               
"""                                                                         
import mappyfile                                                                
                                                                                
mappyfile.loads(layer)

the code sample above failes and complains about !
But this is still a valid EXPRESSION.
By the way thanks for this lib. ;)

See also:
http://mapserver.org/fr/mapfile/expressions.html#logical-expressions

While this clearly ok:
https://s.geo.admin.ch/74d42d8231

When parsing a map file with includes on windows, parsing fails as it looks in the parent directory instead of the current one,

Fixed by #54

In our of our projects our core workflow is to generate a mapfile based on various system configurations (YAML, JSON, etc.). It would be valuable to have mappyfile be able to save a mapfile from a Python dictionary. Example of possible workflow:

import mappyfile

my_mapfile = {
    'name': 'foo'
}
mappyfile.dumps(my_mapfile)  # to string
mappfile.dump(my_mapfile, '/tmp/foo.map')  # to file

    LAYER
        NAME 'regexp-example'
        FILTERITEM 'placename'
        FILTER \hotel\
    END

Produces:

UnexpectedInput: No token defined for: '\' in '\\hote' at line 5

See test_regex in tests/test_snippets.py

Last week I installed 'mappyfile' from pip and ran a ' mappyfile.load(file)' command. The strack trace threw this back at me. I checked that directory...the error doesn't lie. Indeed there is no 'mapfile.g'.

Can someone verify the pip install package is ok....and if it isn't...is the .whl ok?

IOError: [Errno 2] No such file or directory: 'C:\Users\ASimmons\.virtualenvs\gis_test\lib\site-packages\mappyfile\mapfile.g'

    OUTPUTFORMAT
        NAME "shapezip"
        DRIVER "OGR/ESRI Shapefile"
        TRANSPARENT FALSE
        IMAGEMODE FEATURE
    END

Gives the following parser error:

Unexpected token Token(__FEATURE24, u'FEATURE')

Great work! Thanks!

This is mostly question/feature request.

I wonder if it would be possible to parse expressions to a deeper level? Like parsing the expression itself to a tree json structure. It seems to me that mappyfile is already doing some validation on the expressions. Comparison/Regex/List type expressions are relaitvely simple compared to Mapserver style expressions.

The rationale: I'm making a transformer from mapfile to mapbox json style specs[1] and would need to convert expressions to filters. I'm using exclusively "Mapserver" expressions [2] in my mapfiles.

Any idea, suggestion is welcome!
[1] https://www.mapbox.com/mapbox-gl-js/style-spec/
[2] https://mapserver.org/mapfile/expressions.html#mapserver-expressions

MapServer allows for OR or ||, AND or &&, and NOT or !

http://www.mapserver.org/mapfile/expressions.html#logical-expressions

Currently only OR, AND and NOT are supported. The following syntax will fail:

    CLASS
        EXPRESSION ("[style_class]" = "10" || "[style_class]" = "20")
    END

See test_or_expressions, test_and_expressions, and test_not_expressions in in tests/test_snippets.py

Can you help me about this error?
`Traceback (most recent call last):
File "/Users/n2software/Documents/djangotest/testpro/maptest.py", line 11, in
ast = p.parse_file('default.map')
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/mappyfile/parser.py", line 181, in parse_file
return self.parse(text, fn=fn)
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/mappyfile/parser.py", line 194, in parse
tree = self.lalr.parse(text)
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/lark/lark.py", line 223, in parse
return self.parser.parse(text)
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/lark/parser_frontends.py", line 36, in parse
return self.parser.parse(token_stream, *[sps] if sps is not NotImplemented else [])
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/lark/parsers/lalr_parser.py", line 68, in parse
for i, token in enumerate(stream):
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/lark/lexer.py", line 250, in lex
for x in l.lex(stream, self.root_lexer.newline_types, self.root_lexer.ignore_types):
File "/Users/n2software/Documents/djangotest/testenv/lib/python3.6/site-packages/lark/lexer.py", line 105, in lex
raise UnexpectedCharacters(stream, line_ctr.char_pos, line_ctr.line, line_ctr.column, state=self.state)
lark.exceptions.UnexpectedCharacters: No terminal defined for 'b' at line 1 col 1

b'\x08Turkiye6\xc3\x89|B\xc3\x82\xc3\xad`

This is the code
import mappyfile
from django.core.serializers import json
from mappyfile.parser import Parser
from mappyfile.transformer import MapfileToDict
from mappyfile.pprint import PrettyPrinter
from lark import UnexpectedToken
from mappyfile.validator import Validator

p = Parser()
m = MapfileToDict()
ast = p.parse_file('default.map')
d = m.transform(ast)
print(json.dumps(d, indent=4))

On pypi only the whl is present.
Offering a source package would be a must (including tests).

Also whl is missing license file.

On the same side, having the release set in correlation here in github help to grab patches if needed.

It looks (if I'm not doing something wrong way), that loads don`t work well with relative paths in include. On debian.

mpf=mappyfile.open("cesky-rudolec/cesky-rudolec.map")
Works well
but

mpf=mappyfile.load(f)                                         
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/mappyfile/utils.py", line 59, in load
    ast = p.load(fp)
  File "/usr/local/lib/python2.7/dist-packages/mappyfile/parser.py", line 163, in load
    return self.parse(text)
  File "/usr/local/lib/python2.7/dist-packages/mappyfile/parser.py", line 185, in parse
    text = self.load_includes(text, fn=fn)
  File "/usr/local/lib/python2.7/dist-packages/mappyfile/parser.py", line 70, in load_includes
    raise ex
IOError: [Errno 2] No such file or directory: '/gis/mapserv/../include/include_head.map'

Hello,

I've got a Mapfile whith the line "FILTER ( type LIKE '611_%' )" in it and when I try do parse the file I've got this error message :
ERROR:root:Unexpected token Token(NAME, u'LIKE') at line 8283, column 18.
Expected: set([u'__AND13', u'__NE22', u'__IN21', '__TILDE', '__PLUS', '__EQUAL', '__LESSTHAN', '__MORETHAN', '__RPAR', '__ANONSTR_19', '__ANONSTR_18', '__LPAR', '__ANONSTR_20', u'__OR11', '__ANONSTR_12', u'__EQ23', '__ANONSTR_17', '__ANONSTR_16', '__ANONSTR_15', '__ANONSTR_14'])
Context:
Traceback (most recent call last):
File "C:\OSGEO41\apps\Python27\lib\site-packages\mappyfile\parser.py", line 99, in parse
ast = self.earley.parse(text)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\lark.py", line 193, in parse
return self.parser.parse(text)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parser_frontends.py", line 115, in parse
return self.parser.parse(tokens)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parsers\earley.py", line 209, in parse
column = scan(i, token, column)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parsers\earley.py", line 198, in scan
raise UnexpectedToken(token, expect, stream, i)
UnexpectedToken: Unexpected token Token(NAME, u'LIKE') at line 8283, column 18.
Expected: set([u'__AND13', u'__NE22', u'__IN21', '__TILDE', '__PLUS', '__EQUAL', '__LESSTHAN', '__MORETHAN', '__RPAR', '__ANONSTR_19', '__ANONSTR_18', '__LPAR', '__ANONSTR_20', u'__OR11', '__ANONSTR_12', u'__EQ23', '__ANONSTR_17', '__ANONSTR_16', '__ANONSTR_15', '__ANONSTR_14'])
Context:
ERROR:root:Parsing with LALR and Earley unsuccessful
Traceback (most recent call last):
File "", line 1, in
File "C:\OSGEO41\apps\Python27\lib\site-packages\mappyfile\utils.py", line 9, in load
ast = p.parse_file(fn)
File "C:\OSGEO41\apps\Python27\lib\site-packages\mappyfile\parser.py", line 76, in parse_file
return self.parse(text, fn=fn)
File "C:\OSGEO41\apps\Python27\lib\site-packages\mappyfile\parser.py", line 99, in parse
ast = self.earley.parse(text)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\lark.py", line 193, in parse
return self.parser.parse(text)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parser_frontends.py", line 115, in parse
return self.parser.parse(tokens)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parsers\earley.py", line 209, in parse
column = scan(i, token, column)
File "C:\OSGEO41\apps\Python27\lib\site-packages\lark\parsers\earley.py", line 198, in scan
raise UnexpectedToken(token, expect, stream, i)
lark.common.UnexpectedToken: Unexpected token Token(NAME, u'LIKE') at line 8283, column 18.
Expected: set([u'__AND13', u'__NE22', u'__IN21', '__TILDE', '__PLUS', '__EQUAL', '__LESSTHAN', '__MORETHAN', '__RPAR', '__ANONSTR_19', '__ANONSTR_18', '__LPAR', '__ANONSTR_20', u'__OR11', '__ANONSTR_12', u'__EQ23', '__ANONSTR_17', '__ANONSTR_16', '__ANONSTR_15', '__ANONSTR_14'])
Context:

Thank you for your work.
Sébastien Barre.

I have a mapfile with multiple INCLUDE statements:

import mappyfile

c="""
MAP
    NAME "test_map"
    STATUS ON
    EXTENT 2550300 1120800 2688900 1247400
    UNITS METERS
    PROJECTION
        "init=epsg:2056"
    END
	LAYER
		INCLUDE "a.map"
		INCLUDE "b.map"
        NAME "test_layer"
	END
END
"""
mf = mappyfile.loads(c,expand_includes=False)
print mappyfile.dumps(mf)`

which dumps as

MAP
    NAME "test_map"
    STATUS ON
    EXTENT 2550300 1120800 2688900 1247400
    UNITS METERS
    PROJECTION
        "init=epsg:2056"
    END
    LAYER
        INCLUDE "b.map"
        NAME "test_layer"
    END
END

Only the second INCLUDE is dumped. Any idea why?

The following forms parse correctly:

    STYLE
        PATTERN 5.0 5.0 END 
    END

    STYLE 
        PATTERN 
            5.0 5.0 
        END 
    END

However the two styles below do not parse correctly, but are valid types when using the MapServer C parser:

STYLE PATTERN 5 5 END END

    STYLE 
        PATTERN 
            5
            5 
        END 
    END

See test_style_pattern3 and test_style_pattern4 in tests/test_snippets.py

import mappyfile
mappyfile.loads('EXPRESSION ([typ_nr] = 1)')

this expression above will fail.

http://mapserver.org/mapfile/expressions.html#arithmetic-expressions-that-return-a-logical-value

Need to set cwd to an empty string rather than None otherwise:

TypeError: object of type 'NoneType' has no len()

When trying to run this example from the mappyfile docs:

import mappyfile
s = '''MAP NAME "TEST" END'''

d = mappyfile.loads(s)
fn = r"C:\Data\mymap.map"
mappyfile.save(d, fn)

I get the following error:

AttributeError: module 'mappyfile' has no attribute 'loads'

I get the same error when trying any other command (open, load, etc.) . I am using Ubuntu 16.04 and Python 3.5. Is it something I am doing wrong?

See the examples at http://mapserver.org/mapfile/expressions.html#string-expressions-that-return-a-logical-value

EXPRESSION ( "[building]" eq NULL)

import mappyfile

c="""
CLASS
  EXPRESSION {2_Klass,Rte2etr}
  STYLE
    ANGLE 360
    COLOR 255 255 26
    OPACITY 100
    WIDTH 1
  END
END
"""
mappyfile.loads(c)

This will raise:

lark.common.UnexpectedToken: Unexpected token Token(NAME, '_Klass') at line 3, column 15.
Expected: set(['__COMMA', '__RBRACE'])
Context: <no context>

The same without the 2 works fine.

layer = """
LAYER
  CLASSITEM "objval"
  CONNECTION "user=user password=password host=host port=5432 dbname=mydb"
  CONNECTIONTYPE POSTGIS
  DATA "the_geom from vec200_railway using unique gtdboid using srid=21781"
  GROUP "Vector200"
  METADATA
    "wms_srs" "EPSG:21781 EPSG:2056 EPSG:4326 EPSG:3857 EPSG:3034 EPSG:3035 EPSG:4258 EPSG:31287 EPSG:25832 EPSG:25833 EPSG:31467 EPSG:32632 EPSG:32633"
    "wms_group_title" "vec200"
    "wms_title" "Vector200 railway"
    "gml_featureid" "gtdboid"
    "ows_include_items"   "all"
    "ows_exclude_items"   "bgdi_modified,bgdi_created,bgdi_modified_by,bgdi_created_by"
  END
  TEMPLATE "ttt"
  NAME "ch.swisstopo.vec200-transportation-railway"
  STATUS OFF
  TEMPLATE "ttt"
  TYPE LINE
  UNITS METERS
  CLASS
    NAME "NS_Bahn_NS_BahnAuto"
    EXPRESSION /NS_Bahn|NS_BahnAuto/
    STYLE
      ANGLE 360
      COLOR 230 0 0
      OPACITY 100
      WIDTH 3
    END
  END
END
"""
import mappyfile

mappyfile.loads(layer)

It fails. See: http://mapserver.org/fr/mapfile/expressions.html#list-expressions

{NS_Bahn,NS_BahnAuto} does work but /NS_Bahn|NS_BahnAuto/ does not.

List expressions seem to be better anyway, but it is still a valid expression.

I've tested this nice project. I commented out an "END" at the middle of a map file and got "recursion depth" error, the trace log is very long, last few lines (the same messages are repeated):
File "/usr/lib/python2.7/functools.py", line 87, in lt
return mycmp(self.obj, other.obj) < 0
File "/usr/local/lib/python2.7/dist-packages/lark/parsers/earley.py", line 257, in _compare_drv
c = _compare_drv(t1, t2)
File "/usr/local/lib/python2.7/dist-packages/lark/parsers/earley.py", line 247, in _compare_drv
_resolve_ambig(tree1)
File "/usr/local/lib/python2.7/dist-packages/lark/parsers/earley.py", line 267, in _resolve_ambig
best = min(tree.children, key=cmp_to_key(_compare_drv))
File "/usr/lib/python2.7/functools.py", line 87, in lt
return mycmp(self.obj, other.obj) < 0
File "/usr/local/lib/python2.7/dist-packages/lark/parsers/earley.py", line 257, in _compare_drv
c = _compare_drv(t1, t2)
File "/usr/local/lib/python2.7/dist-packages/lark/parsers/earley.py", line 234, in _compare_drv
if not (isinstance(tree1, Tree) and isinstance(tree2, Tree)):
RuntimeError: maximum recursion depth exceeded while calling a Python object

I attach the file (see the commented line at 3000) (renamed to txt because of the github limitation)
x201703.txt

I have the following issue, let's take for instance this example.

import mappyfile                                                                
from mappyfile.pprint import PrettyPrinter                                      
                                                                         
def _pprint(d, indent=2):                                                       
    pp = PrettyPrinter(indent=indent)                                           
    return pp.pprint(d)                                                         
                                                              
c = """                                                                         
CLASS                                                                           
    EXPRESSION (('[agglo_klas]'='K') OR ('[agglo_klas]'='M'))                   
END                                                                             
"""
                                                       
m = mappyfile.loads(c)                                                          
print _pprint(m)                                                                

m = mappyfile.loads(_pprint(m))                                                 
print _pprint(m)

This will result in:

CLASS
EXPRESSION (( (( '[agglo_klas]' = 'K' )) or (( '[agglo_klas]' = 'M' )) ))
END
CLASS
EXPRESSION ((( ((( '[agglo_klas]' = 'K' ))) or ((( '[agglo_klas]' = 'M' ))) )))
END

Adding more and more parenthesis after each operation. If I want to use it as an automatic linter for my mapfiles, I want the structure to remain stable between each iteration. Any way we can tune that somewhere?

FYI I was fine with the first EXPRESSION and didn't need to have all these parenthesis added in the first place.

If you have a PATTERN element in your mapfile like:

PATTERN 6 4 2 4 6 END

mappyfile throws a syntax error for an unrelated line number, this makes debugging very hard. Unfortunately MapServer and shp2img both eat the line silently (and I guess drop the last number) so are also no help in debugging.

A better error message or if the syntax error at least gave the right line it would help.

There are a couple of expressions that aren't catered for. See tests in test_expressions.py.

• test_divide_expression
• test_multiply

In the packages setup.py file the "lark-parser" version is not fixed to a number. So if we use it now pip installs the newest version of "lark-parser" which is 0.5.1

This has a problem with this call:
https://github.com/geographika/mappyfile/blob/master/mappyfile/parser.py#L28

It seems that "lark-parser" does not support parameter "earley__all_derivations" any longer in newest version. So parsing has been corrupted in this combination.

If I install manually version 0.4.1 of "lark-parser" everything works like expected:
pip install "lark-parser==0.4.1"

Suggestion is to set version of lark-parser fix to 0.4.1 in setup.py

Given a mapfile containing:

QUERYMAP
   COLOR 255 255 0
   SIZE -1 -1
   STATUS OFF
   STYLE HILITE
 END

Parsing fails at the STYLE line (while reporting an unrelated line number). This appears to be a correct formulation according to the MapServer manual.

It looks like a grammar error but I don't want to mess with that ;-)

MAP
	CONFIG "PROJ_LIB" "projections"
END

After pretty printing becomes:

MAP
    config "PROJ_LIB" "projections"
END

given a line like:

EXPRESSION ([Cluster:FeatureCount] == "1")

parsing fails with No token defined for ':'

Escaped strings e.g. "National "hero" statue" fail to parse. At one point these were working, but
not sure what was changed to break these. Broken tests are:

• test_expressions.py - test_escaped_string
• test_pprint.py - test_already_escaped

Usually I find it useful to flake8 linter to keep a clean code and nice code convention accross the project.
Do you think you'd like to have that in your project?

Doc:
http://flake8.pycqa.org/en/latest/

If you agree, I could make a PR for that.

You don't need to enforce all the rules, and you may want to keep our own style in some part of the code, but this can be tuned on demand.

There are currently two ambiguities that remain in the grammar.

1. SYMBOL can in some cases be used as a composite type, and sometimes as an attribute. E.g.

SYMBOL
  NAME "circle"
  TYPE ellipse
  FILLED true
  POINTS
    1 1
  END
END

And sometimes it can be used as a simple keyword / value pair e.g.

SYMBOL "symbol_name
SYMBOL 0 # reference a symbol from a SYMBOLSET

Everything currently works fine for cases such as:

SYMBOL"barb_warm"

But in cases without quotes, such as the following, fail:

SYMBOL barb_warm

Not sure if anything can be done about this. If not then these will always need to be quoted.
There is a test for this in test_snippets.py - test_symbol_style2().

A possible solution is to have an unquoted string type, a KEYWORD type which only accepts letters (case-insensitive)
for attribute names, and a more permissive type that accepts underscores, numbers, and hyphens that could
be used for attribute values.

2. STYLE - this is typically a class such as:

STYLE
  SYMBOL 'hatch-test'
  COLOR 255 0 0
  ANGLE [MYROTATE]
  SIZE 4.0
  WIDTH 3.0
END

However it is also used in a two other classes as an attribute e.g. in a SCALEBAR and in a QUERYMAP. E.g.

SCALEBAR
    STYLE 0
END

import mappyfile                                                                
                                                                                
c = """                                                                         
INCLUDE "includes/mymapfile.map"                                                
"""                                                                             
                                                                                
mappyfile.loads(c, expand_includes=False) 

is interpreted as a string and not a path.

lark.common.UnexpectedToken: Unexpected token Token(STRING1, '"includes/mymapfile.map"') at line 2, column 8.
Expected: set([u'__LABEL34', u'__CONFIG30', u'__LAYER35', u'__LEADER36', u'__REFERENCE41', u'__COMPOSITE29', u'__METADATA4', u'__POINTS0', u'__CLUSTER28', u'__CLASS8', u'__SYMBOL6', u'__VALIDATION5', u'__STYLE44', u'__OUTPUTFORMAT39', u'__FEATURE9', u'__QUERYMAP40', u'__PATTERN1', u'__FONTSET31', u'__SCALETOKEN43', u'__INCLUDE32', u'__PROJECTION2', u'__GRID7', u'_END', u'__WEB45', u'__MAP38', u'__LEGEND37', u'NAME', u'__JOIN33', u'__SCALEBAR42', u'__VALUES3'])
Context: <no context>

import mappyfile

c="""
CLASS
  NAME "Autobahn"
  EXPRESSION ( '[construct]' eq 'keine' ) or ( '[contruct]' eq 'twice' )
  STYLE
    ANGLE 360
    COLOR 0 0 0
    OPACITY 100
    WIDTH 6
  END
END
"""
mappyfile.loads(c)

I found another error with expressions.

Expected: set([u'__LABEL34', u'__CONFIG30', u'__LAYER35', u'__REFERENCE41', u'__LEADER36', u'__COMPOSITE29', u'PATH', u'__METADATA4', u'__POINTS0', '__LBRACE', u'__CLUSTER28', u'__CLASS8', u'__SCALEBAR42', u'__JOIN33', u'__VALIDATION5', u'SIGNED_INT', u'__NOT10', u'SIGNED_FLOAT', u'__STYLE44', u'__FEATURE9', u'STRING2', u'STRING3', u'__PATTERN1', u'__FONTSET31', u'__SCALETOKEN43', u'__INCLUDE32', u'__WEB45', u'AUTO', u'__PROJECTION2', u'__GRID7', u'__OUTPUTFORMAT39', u'_END', '__LPAR', u'__MAP38', u'__LEGEND37', u'NAME', '__LSQB', u'REGEXP2', u'__QUERYMAP40', u'REGEXP1', u'__SYMBOL6', u'RUNTIME_VAR', u'__VALUES3', '__BANG', u'STRING1'])
Context: <no context>

changing the prathesis to ( '[construct]' eq 'keine' or '[contruct]' eq 'twice' ) solves the issue.

It think there is still some work to be done with expressions in general.

(( '[construct]' eq 'keine') or ( '[contruct]' eq 'twice' )) also does

At the moment logging is imported directly in the different modules, making it impossible to shut him up.

The simple solution would be to create a logger named mappyfile and then import it everywhere.

With a simple note in a script:

logging.getLogger('mappyfile').setLevel(logging.WARNING)

you can set a custom logging level.

  OUTPUTFORMAT
    NAME "shapezip"
    DRIVER "OGR/ESRI Shapefile"
    TRANSPARENT FALSE
    FORMATOPTION "STORAGE=memory"
    FORMATOPTION "FORM=zip"
    FORMATOPTION "SPATIAL_INDEX=YES"
  END # OUTPUTFORMAT

Results in:

OUTPUTFORMAT
    NAME "shapezip"
    DRIVER "OGR/ESRI Shapefile"
    TRANSPARENT FALSE
    FORMATOPTION "SPATIAL_INDEX=YES"
END

This is an issue with the transformer rather than the parser.