Code Monkey home page Code Monkey logo

coast's Introduction

coAST

coAST is a universal abstract syntax tree that allows to easily analyze each programming language. Especially adding new languages should be easy and generic.

Goals

  1. Describe languages using theoretical components, aimed at human comprehension, so that further understanding of concepts used by a language can be obtained by reading online resources rather than code.

  2. Provide multiple usable levels of parse-ability, so that a file can be accurately split into parts which are not yet parse-able, or the use case has no benefit in parsing, and the parts may be modified and re-joined into an otherwise semantically equivalent file.

Performance and algorithmic beauty are not goals. Reversibility, like augeas, is not a goal, as that requires a Context Sensitive Tree.

To achieve the first goal, the primary output of this repository is a static website which allows the reader to understand the definitions contained here, and link to other online resources where more information can be obtained.

Links to Wikidata, Antlr definitions, E(BNF) files, example files, will be integral components of the definitions here.

Terminology used to describe language components will be consistent across languages where-ever possible, and defer to terminology used in academic literature or study guides, to make these definitions more accessible and useful to students of language theory.

Stages

  1. Organically grow a human readable fact based database of any syntax, stored in YAML files, covering any language from large and complicated programming languages down to strings like a URL, especially focusing on style description which describe a subset of a language.

  2. Create programs to load these definitions and convert input files into a universal AST, primarily for building a test suite to verify the language definitions are able to parse files at useful levels of detail, again focusing on style-defined subsets of languages which are easier and also more useful.

    These programs may use existing parsers, by converting the coAST definitions into metasyntax used by other parsing toolkits, such as BNF and derivatives, Antlr .g4, and augeas.

  3. Standardise the definition schema once a sufficiently large number of language definitions have been adequately verified to determine the schema is able to usefully describe most concepts found in commonly used grammars.

Phases

These phases will be overlapping slightly.

Phase 1: Replace coala language definitions

The language definitions found at https://github.com/coala/coala/tree/master/coalib/bearlib/languages will be manually added as language definitions here, growing the schema as necessary. Once the import of facts is complete, a generator will create the coala language definitions from a snapshot of the coAST language definitions, putting the collated coAST definitions into use.

Phase 2: Import other language definitions

There are many other collections of language definitions. Initially the coAST definitions will only link to these external resources, and then in the second phase those external grammars will be converted into coAST facts, using batch import tools or manually where necessary.

In this phase tools to convert the coAST definitions into other syntax will be needed, to round trip the language definitions, providing verification that the imports are complete, or that partial definitions allow correct partial parsing of those languages where complete parsing is too complex.

Phase 3: Create language style definitions

Create declarative descriptions of common styles, such as the Google Python coding guidelines, and Airbnb JavaScript style.

The schema for describing styles will borrow from the coala aspects definitions, and should allow users to define their own custom styles, however the priority will be accurately describing well established style guides, and important features of commonly used linters of various languages.

Phase 4: Replace coala aspects

coala aspects development is driven by the needs of users, the complexity of bears, and pre-existing implementation choices of coala.

For avoid these influences causing incorrect design decisions in coAST, importing of aspects will not be considered until after style definitions are in place.

Authors

coAST is maintained by the coala community. Contact us on gitter!

Licenses

The facts in this repository are inherently public domain, and are explicitly released under the CC-0 license. https://creativecommons.org/publicdomain/zero/1.0/

The website templates and assets included in this repository are released under the Creative Commons Share-alike license 4.0. https://creativecommons.org/licenses/by-sa/4.0/

Any code in this repository is to be release under the MIT license. https://opensource.org/licenses/MIT

coast's People

Contributors

abhishalya avatar cliu13 avatar cubetastic33 avatar dhruvsomani avatar jayvdb avatar kiy4h avatar koru1130 avatar naveenaidu avatar naveentvelu avatar nneast avatar p-murli avatar rajgoesout avatar razvanmitrica avatar seeeturtle avatar serekkiri avatar shakeelsamsu avatar sheshank-s avatar siddhpant avatar sils avatar suggoitanoshi avatar tanmaysachan avatar tushar-rishav avatar why-does-ie-still-exist avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

coast's Issues

Netlify is failing

Netlify has been failing due to some inconsistencies in the Language file.
The problem as I read the logs are in the file

  1. Pascal.yaml - Datatypes haven't been created
  2. VisualBasic.yaml - A non string value present in the keywords.

The logs:

  1. https://app.netlify.com/sites/coast/deploys/5bf80b4d9a063f48ba6cb4c6
  2. https://app.netlify.com/sites/coast/deploys/5bfea5fa05c41723bdd7edc0

The error messages:

These messages occur after one of them fixed. I mean they occur one after another

WARNING Non-string values found in array (field: keywords, instance: Pascal, model: Language): True

ERROR	Error: Failed to build project. Run Statik in verbose mode (-v) to see additional traceback information about this error. (in file "/opt/build/repo/data/Language/Pascal.yaml"). Additional error detail: '1' is not in list
WARNING	Non-string values found in array (field: keywords, instance: VisualBasic, model: Language): True

ERROR	Error: Failed to build project. Run Statik in verbose mode (-v) to see additional traceback information about this error. (in file "/opt/build/repo/data/Language/VisualBasic.yaml"). Additional error detail: '1' is not in list

Add Travis CI to run latest coala

The repository has a .coafile, and GitMate does check it, but it is better to have Travis CI also checking it.

It needs to use the latest coala, from git

Remove `:` after a delimter section in Language file

Add styling to the site

We need to add some design to the site generated to make it look more beautiful.

coala uses materialize css for designing the webpages. You can find some inspiration in the other repos which render html pages.

The role of keywords

keywords is covering too much things: statements, expressions(including operators), types etc.
I suggest to split them into several pieces like the delimiters.

Add .coafile

This needs a .coafile checking the yaml and templates.

README.md: Update grammar at line 97

Under Phase 4, it says:
For avoid these influences causing incorrect design decisions in coAST ...

For avoid doesn't look quite right and hence is to be updated.

A good newcomer issue.

Strange build failure

Strange build failure. Rebuilding without cache solved it...

8:32:21 PM: --- Logging error ---
8:32:21 PM: Traceback (most recent call last):
8:32:21 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 997, in emit
8:32:21 PM:     self.flush()
8:32:21 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 977, in flush
8:32:21 PM:     self.stream.flush()
8:32:21 PM: BlockingIOError: [Errno 11] write could not complete without blocking
8:32:21 PM: Call stack:
8:32:21 PM:   File "/opt/buildhome/python3.6/bin/statik", line 11, in <module>
8:32:21 PM:     sys.exit(main())
8:32:21 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/cmdline.py", line 211, in main
8:32:21 PM:     error_context=error_context
8:32:21 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/generator.py", line 15, in generate
8:32:21 PM:     return project.generate(output_path=output_path, in_memory=in_memory)
8:32:21 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/project.py", line 148, in generate
8:32:21 PM:     self.db.shutdown()
8:32:21 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 374, in shutdown
8:32:21 PM:     clear_tracked_globals()
8:32:21 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 61, in clear_tracked_globals
8:32:21 PM:     logger.debug('Clearing tracked global: %s' % name)
8:32:21 PM: Message: 'Clearing tracked global: FileExtensionLanguage'
8:32:21 PM: Arguments: ()
8:32:21 PM: 2018-10-27 13:32:12,866	statik.database	DEBUG	Clearing tracked global: Keyword
8:32:21 PM: --- Logging error ---
8:32:21 PM: Traceback (most recent call last):
8:32:21 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 997, in emit
8:32:21 PM:     self.flush()
8:32:21 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 977, in flush
8:32:21 PM:     self
8:32:21 PM: .stream.flush()
8:32:21 PM: BlockingIOError: [Errno 11] write could not complete without blocking
8:32:22 PM: Call stack:
8:32:22 PM:   File "/opt/buildhome/python3.6/bin/statik", line 11, in <module>
8:32:22 PM:     sys.exit(main())
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/cmdline.py", line 211, in main
8:32:22 PM:     error_context=error_context
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/generator.py", line 15, in generate
8:32:22 PM:     return project.generate(output_path=output_path, in_memory=in_memory)
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/project.py", line 148, in generate
8:32:22 PM:     self.db.shutdown()
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 374, in shutdown
8:32:22 PM:     clear_tracked_globals()
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 61, in clear_tracked_globals
8:32:22 PM:     logger.debug('Clearing tracked global: %s' % name)
8:32:22 PM: Message: 'Clearing tracked global: Keyword'
8:32:22 PM: Arguments: ()
8:32:22 PM: 2018-10-27 13:32:12,866	statik.database	DEBUG	Clearing tracked global: AliasLanguage
8:32:22 PM: --- Logging error ---
8:32:22 PM: Traceback (most recent call last):
8:32:22 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 997, in emit
8:32:22 PM:     self.flush()
8:32:22 PM:   File "/usr/lib/python3.6/logging/__init__.py", line 977, in flush
8:32:22 PM:     self.stream.flush()
8:32:22 PM: BlockingIOError: [Errno 11] write could not complete without blocking
8:32:22 PM: Call stack:
8:32:22 PM:   File "/opt/buildhome/python3.6/bin/statik", line 11, in <module>
8:32:22 PM:     sys.exit(main())
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/cmdline.py", line 211, in main
8:32:22 PM:     error_context=error_context
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/generator.py", line 15, in generate
8:32:22 PM:     return project.generate(output_path=output_path, in_memory=in_memory)
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/project.py", line 148, in generate
8:32:22 PM:     self.db.shutdown()
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 374, in shutdown
8:32:22 PM:     clear_tracked_globals()
8:32:22 PM:   File "/opt/buildhome/python3.6/lib/python3.6/site-packages/statik/database.py", line 61, in clear_tracked_globals
8:32:22 PM:     logger.debug('Clearing tracked global: %s' % name)
8:32:22 PM: Message: 'Clearing tracked global: AliasLanguage'
8:32:22 PM: Arguments: ()
8:32:22 PM: 2018-10-27 13:32:12,867	statik.database	DEBUG	Clearing tracked global: Alias
8:32:22 PM: --- Logging error ---
8:32:22 PM: Traceback (most recent call last):
8:32:22 PM:   File "/usr/lib/python3.6/logging/__init__.py", linCaching artifacts

Netlify failing on pip install

9:08:24 PM: Finished restoring cached pip cache
9:08:24 PM: Collecting git+https://github.com/thanethomson/statik (from -r requirements.txt (line 1))
9:08:24 PM:   Cloning https://github.com/thanethomson/statik to /tmp/pip-req-build-j8yeyb_j
9:08:25 PM:     Complete output from command python setup.py egg_info:
9:08:25 PM:     error in statik setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'"

Fix Documentation Error in _all.yml

The word currently has been repeated twice on line 10 of the file _all.yml

# 'block' currently currently implies a new 'scope', however we may need to

Label:- difficulty/newcomer

Remove duplicate datatype files

There are some datatype files which are present, for example, long_java and long_d.

They should be removed, since duplicates are generally not required.

The new names of datatypes should allow it to be clearly differentiated from others and should be related to the concerned datatype.

Import languages from Pygments

Create a script, any language, which must be checked into /tools/ in this repo and be run in Travis CI, which creates data files for any languages that are in Pygments but are missing from coAST.

This would replace coala/coala#4556 .

Try to include as much information from Linguist as possible, enhancing the local schema as needed.

a relevant WIP is coala/coala#3162

Note the hard part of this is merging language versions to avoid essentially duplicate languages. See some of the logic from .misc/pygments_coalang_import.py in that WIP .

AST vs CST

It is hard to auto-fix code if you only have an AST. Vital information has been dropped.
Also many of the generic bears rely on information typically not present in an AST.

Do you actually want a universal CST also?

Add antlr language .g4

The keywords for both the versions are same and there is just a change of extensions where the newer version automatically constructs the parse tree.
So i will be sending PR on this, if its okay.

Grammar file produces a hyperlink even when no value is given to it in the Language files

Grammer file produces a hyperlink to the respective grammar files when we provide the entry to it in the language file.

Example:- C.yaml
grammar_file: https://github.com/antlr/grammars-v4/blob/master/c/C.g4

After the site is deployed, the results are rendered as: https://coast.netlify.com/c/
As you see here a hyperlink is produced for the grammar file.

But

When the grammar file field is empty or not mentioned in the language file such as Bash.yaml or Swift.yaml where the grammar file is not present.

When these files get rendered, they display the text None which hyperlinks to a page which do not exist.

Grammar File None

This needs to be fixed. Instead of showing None with a hyperlink. It should only show None without any hyperlinks.

label: difficulty/low

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.