octopus-platform / joern Goto Github PK

A robust parser for C/C++ storing abstract syntax trees, control flow graphs and program dependence graphs in a neo4j graph database.

Home Page: http://mlsec.org/joern

License: GNU Lesser General Public License v3.0

Shell 0.28% Python 7.37% ANTLR 0.73% Java 87.24% C 0.44% C++ 0.01% Groovy 3.00% PHP 0.93%

joern's People

Contributors

Stargazers

Watchers

Forkers

a0x77n blame1 detectivebag pombredanne amanone rgbkrk d4nnyk nulng chubbymaggie luiseduardohdbackup jcarlson23 agelastic minlex mvladk movingname lucabongiorni subailong annamalai-nr tvondra amesianx suto savaki vlad902 him-28 terry2012 infiboy vvalter waitingcheung ideadapt nccgroup skypacer210 tdjackey d0c-s4vage lhsail yangke oopsla2015 minhyuk qhnetspirit anukat2015 zdzhjx yhvhvx cherry-wb cpx0rpc wflk clccc antoinehus xujiesi brianwrf st12345 qingkunm tuantmtb evonide timhemel machiry svanteschubert gitcollect sigma-random mbrukman yuede sandikast lingfennan qianlitayunhai amlweems josh200501 wsxarcher vaioco turned2670 michicho idl3r rongqinglee ianmadlenya tiandiyixian mewbak ajgappmark houcy tumblr1978 jjiad12 juhakivekas pauls658 airbrid rsalesc aalhuz singleghost rojaster renesugar thebabush robbiejones96 thor509 coder-chenzhi jack23912 fjsnogueira bingslient 8589 wawava wal613 song-li droc egor-bogomolov waterdemo sanying123

joern's Issues

Unsanitized probably doesn't grub arrays

Query like getArguments('f', '0').unsanitized({it,s -> it._().codeContains('$a'))}) (i.e. unsanitzed shouldn't filter anything)

With following signature of f:

void f(int *a);

grubs normally:

void g(int *a) {
  f(a);
}

but doesn't work with:

void g(int *a[]) {
  f(a[0]);
}

code-attribute of ArgumentLists

The .code-attribute of ArgumentLists is not correct.

Documentation update

Are you planning to update docs and python-joern's docs? It's now about 2 years old
There's some cases that really can make your life miserable and spend lot of your time like:
Not working .unsanitized{ it.isCheck( paramName ) }; working: .unsanitized{ it -> it.isCheck( paramName ) } (that also can be due to groovy/gremlin/neo4j updates, just guess by numerous attempts)
Not working getArguments('(f1 OR f2)', '0'); possible workaround: OR(getArguments('f1', '0'), getArguments('f2', '0')
And probably much more stuff like this.

Warning in nested macro conditionals and Parse error in embeded macro

The following code will not be parsed，and joern will throw two warnings and an error exception .

#if !_LIBC
/* Return non-zero if FD is open.  */
static int
is_open (int fd)
{
# if (defined _WIN32 || defined __WIN32__) && ! defined __CYGWIN__
  /* On native Windows: The initial state of unassigned standard file
     descriptors is that they are open but point to an INVALID_HANDLE_VALUE.
     There is no fcntl, and the gnulib replacement fcntl does not support
     F_GETFL.  */
  return (HANDLE) _get_osfhandle (fd) != INVALID_HANDLE_VALUE;
# else
#  ifndef F_GETFL
#   error Please port fcntl to your platform
#  endif
  return 0 <= fcntl (fd, F_GETFL);
# endif
}
#endif
static void
flush_stdout (void)
{
#if !_LIBC
  int stdout_fd;
# if GNULIB_FREOPEN_SAFER
  /* Use of gnulib's freopen-safer module normally ensures that
       fileno (stdout) == 1
     whenever stdout is open.  */
  stdout_fd = STDOUT_FILENO;
# else
  /* POSIX states that fileno (stdout) after fclose is unspecified.  But in
     practice it is not a problem, because stdout is statically allocated and
     the fd of a FILE stream is stored as a field in its allocated memory.  */
  stdout_fd = fileno (stdout);
# endif
  /* POSIX states that fflush (stdout) after fclose is unspecified; it
     is safe in glibc, but not on all other platforms.  fflush (NULL)
     is always defined, but too draconian.  */
  if (0 <= stdout_fd && is_open (stdout_fd))
#endif
    fflush (stdout);
}
static void
print_errno_message (int errnum)
{
  char const *s;
#if defined HAVE_STRERROR_R || _LIBC
  char errbuf[1024];
# if STRERROR_R_CHAR_P || _LIBC
  s = __strerror_r (errnum, errbuf, sizeof errbuf);
# else
  if (__strerror_r (errnum, errbuf, sizeof errbuf) == 0)
    s = errbuf;
  else
    s = 0;
# endif
#else
  s = strerror (errnum);
#endif
#if !_LIBC
  if (! s)
    s = _("Unknown system error");
#endif
#if _LIBC
  __fxprintf (NULL, ": %s", s);
#else
  fprintf (stderr, ": %s", s);
#endif
}

[phpsupport] Move node and edge strings for unit test cases into individual files

Many node and edge strings for PHP in CSV format are found in TestPHPCSVASTBuilder and TestPHPCSVASTBuilderMinimal. A subset of the former are additionally reproduced in CSVASTSamples. We should move them into individual files and initialize such strings from these files, thus separating program logic and test resources. It will also make it much easier to apply changes to the CSV format, as files can be easily regenerated by some shell script. (This could be made part of rebuildTestDB.sh, or introduced in a similarly-named file such as rebuildTestCSVs.sh.)

Joern and Joern-Tools

I installed Joern and its dependencies as described in the documentation; however, there were issues encountered in using the tool.

The py2neo-gremlin tool you have linked no longer appears compatible with the latest py2neo tool.

Current Link:
https://github.com/fabsx00/py2neo-gremlin/blob/master/py2neo_gremlin/gremlin.py

Latest from py2neo:
https://github.com/nigelsmall/py2neo/blob/a3eae0dae3f18548b4dacfb54df283d5b35c8b3b/py2neo/ext/gremlin/__init__.py

When accessing the code via python-joern per the "Basic Usage" example in the documentation, several errors arise from your gremlin link due to the latest version of py2neo.

For example,

graph_db no longer has the attribute "metadata" (line 13)
Resource does not have _post as an attribute ( line 22)
neo4j does not the attribute _hydrated (line 23)

After updating gremlin.py to reflect the latest version in py2neo/ext/gremlin, it compiled successfully; but, the Basic Usage example still returned no results.

When adding and analyzing "testCode", no data and no relationships were shown in Neo4J (http://localhost:7474/db/data/node/0) when adding testCode; furthermore, when executing "echo 'g.v(0).out()' | joern-lookup -g", no results were returned.
When following the VLC tutorial, no data and no relationships were shown in Neo4J (http://localhost:7474/db/data/node/0) when adding the VLC source code; furthermore, when executing "echo 'g.v(0).out()' | joern-lookup -g", no results were returned.

I would love to utilize your tool, but I don't understand why no results are being returned for any source code analysis after following your documentation.

I appreciate your insight and I look forward to hearing from you.

CSV output

Since a clear interface to the output module exists now, it would be pretty nice to have a CSV output module that stores nodes and edges in a format suitable to be imported using the batch-importer by Michael Hunger at https://github.com/jexp/batch-import/ .
For one thing, we could use this as an interchange format between different graph databases. For another, it might result in a performance gain and outsources the job of efficiently importing into the database to a generic tool.

About the BUG when running CallGraphCreator after indexed the linux kernel file like linux-2.6.10/drivers/usb/media/sn9c102_core.c

Firstly,the thowed exception is pasted here:

Exception in thread "main" java.lang.NullPointerException
at org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.java:54)
at org.apache.lucene.util.StringHelper.intern(StringHelper.java:39)
at org.apache.lucene.index.Term.(Term.java:38)
at org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:643)
at org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1436)
at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1319)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1275)
at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300)
at org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.query(LuceneBatchInserterIndex.java:372)
at output.neo4j.Neo4JBatchInserter.queryIndex(Neo4JBatchInserter.java:69)
at tools.callGraph.CallGraphListener.lookupCallee(CallGraphListener.java:62)
at tools.callGraph.CallGraphListener.resolveCalledFunction(CallGraphListener.java:40)
at tools.callGraph.CallGraphListener.createOutgoingCallGraphEdges(CallGraphListener.java:24)
at tools.callGraph.CallGraphListener.visitFunction(CallGraphListener.java:17)
at tools.callGraph.ImportedFunctionWalker.notifyListenersOfFunction(ImportedFunctionWalker.java:37)
at tools.callGraph.ImportedFunctionWalker.walk(ImportedFunctionWalker.java:29)
at tools.callGraph.CallGraphCreator.main(CallGraphCreator.java:19)

Secondly,I found the exception is about the handling the code like :

linux-2.6.10/drivers/usb/media/sn9c102_core.c

sn9c102_urb_complete (struct urb * urb , struct pt_regs * regs)

if (status) {
DBG(3, "Error in isochronous frame")
(*f)->state = F_ERROR;
continue;
}

This is a "if" statement in linux kernel 2.6.10.
When "joern" index this file,it wrongly handled the "if body" usage code of macro("DBG(...)") included
which caused when you run the main class tools.callGraph.CallGraphCreator
it throws an Null pointer exception because the index misrecognise
DBG(3, "Error in isochronous frame")(*f) as a function
and naively think DBG(3, "Error in isochronous frame") is the functionName.

Here is the debug result when I index the whole linux-2.6.10 directory.
the result is got from the function:

private String getCalleeString(Long callId)

which is called by:

private IndexHits resolveCalledFunction(long callId)

FirstChild of 2830017 is
2830021:DBG ( 3 , "Error in isochronous frame" )
AFTER REP:DBG ( 3 , "Error in isochronous frame" )
2829432:sn9c102_urb_complete (struct urb * urb , struct pt_regs * regs)
linux-2.6.10/drivers/usb/media/sn9c102_core.c

CSV issue

I tried the dev branch with small project(i can share if you need)

First thought was that csv file is generated based on sources but it drops an exception so I guessed that it should work from already generated db

So after generating db(181 nodes & 417 edges) I ran ./joern .joernIndex -outformat csv
and get this file hierarchy:

And all files are look similar to a selected one. So that's quite weird behaviour

Parser fails on convoluted expressions

The parser fails on some valid (but convoluted) expressions such as:

int res = ((int (*)())(*(&function_table[atoi(argv[1])])))(atoi(argv[2]), atoi(argv[3]));

It ends up creating multiple statement nodes that will just contain one-character pieces of the whole expression. Also, it does not report an error.

Full example:

#include <stdio.h>
int sum(int a, int b) {return a+b;}
int mul(int a, int b) {return a*b;}
int main(int argc, char **argv) {
    int (*function_table[2])(int a, int b);
    function_table[0] = sum;
    function_table[1] = mul;
    int res = ((int (*)())(*(&function_table[atoi(argv[1])])))(atoi(argv[2]), atoi(argv[3]));
    printf("result %d\n", res);
    return 0;
}

Building dev branch

Is it intended to build the dev branch or it's not ready yet?

I've tried couple of times with different versions of openjdk(7 & 8). I delete unnecessary info from logs, feel free to ask the full ones

For gradle jar/gradle classes with openjdk-8:

:projects:joern-pluginlib:compileJava .../projects/joern-pluginlib/src/main/java/joern/pluginlib/plugintypes/JoernProjectPlugin.java:4: error: cannot find symbol
import octopus.lib.plugintypes.OctopusProjectPlugin;
                              ^
  symbol:   class OctopusProjectPlugin
  location: package octopus.lib.plugintypes
.../projects/joern-pluginlib/src/main/java/joern/pluginlib/plugintypes/JoernProjectPlugin.java:6: error: cannot find symbol
public class JoernProjectPlugin extends OctopusProjectPlugin {
                                        ^
  symbol: class OctopusProjectPlugin
.../projects/joern-pluginlib/src/main/java/joern/pluginlib/plugintypes/JoernProjectPlugin.java:10: error: cannot find symbol
                setProjectConnector(new JoernProjectConnector());
                ^
  symbol:   method setProjectConnector(JoernProjectConnector)
  location: class JoernProjectPlugin
3 errors

BUILD FAILED

With gradle build & openjdk-8 build fails on tests(due to version maybe?):

:projects:joern-fuzzyc:test

tests.ddg.testDDGCreator > simplestTest FAILED
    java.util.NoSuchElementException at testDDGCreator.java:33

tests.ddg.testDDGCreator > testConverter FAILED
    java.util.NoSuchElementException at testDDGCreator.java:55

tests.argumentTainter.TestArgumentTainter > testDefUseCFGPatcher FAILED
    java.util.NoSuchElementException at TestArgumentTainter.java:65

135 tests completed, 3 failed
:projects:joern-fuzzyc:test FAILED

And finally gradle build/gradle jar/gradle classes & openjdk-7 fails on lambdas at CG.java due to it's unsupported at 7th version

Incorrect handling of try/catch makes evaluating Windows drivers impossible

It seems to me that joern doesn't handle try{}catch{} correctly.

I'm now using the following code:
query = (""
"getArguments('ProbeForRead', '0')"
".match{ it.type == 'Identifier' && it.childNum == '0' }" # This gets the 'identifier' node
".sideEffect { paramName = it.code }"
".reachableCfgNodes()"
)

You can see from the output (below) running against the demo code below that reachableCfgNodes doesn't return either the statements after ProbeForRead that are inside the "try" statement, and it also doesn't return the statements after the "catch" statement. So all that Joern seems to think is reachable when you start from a node inside a "try" statement are the nodes in the "catch" statement.

This prevents Joern from being used against many Windows driver scenarios.

Output:
(n68 {childNum:"0",code:"ntStatus = GetExceptionCode ( )",functionId:14,isCFGNode:"True",location:"24:2:480:509",type:"ExpressionStatement"})

(n118 {code:"EXIT",functionId:14,isCFGNode:"True",type:"CFGExitNode"})

(n66 {childNum:"1",code:"goto Cleanup ;",functionId:14,isCFGNode:"True",location:"25:2:513:525",type:"GotoStatement"})

(n76 {childNum:"6",code:"EXCEPTION_EXECUTE_HANDLER",functionId:14,isCFGNode:"True",location:"21:8:447:471",type:"Statement"})

(n77 {childNum:"5",code:"(",functionId:14,isCFGNode:"True",location:"21:7:446:446",type:"Statement"})

(n78 {childNum:"4",code:"except",functionId:14,isCFGNode:"True",location:"21:1:440:445",type:"Statement"})

(n28 {childNum:"12",code:"return ntStatus ;",functionId:14,isCFGNode:"True",location:"40:1:804:819",type:"ReturnStatement"})

(n30 {childNum:"11",code:"Cleanup :",functionId:14,isCFGNode:"True",location:"39:0:794:801",type:"Label"})

(n75 {childNum:"7",code:")",functionId:14,isCFGNode:"True",location:"21:33:472:472",type:"Statement"})

(n273 {code:"EXIT",functionId:131,isCFGNode:"True",type:"CFGExitNode"})

(n190 {childNum:"8",code:")",functionId:131,isCFGNode:"True",location:"60:33:1323:1323",type:"Statement"})

(n191 {childNum:"7",code:"EXCEPTION_EXECUTE_HANDLER",functionId:131,isCFGNode:"True",location:"60:8:1298:1322",type:"Statement"})

(n145 {childNum:"13",code:"return ntStatus ;",functionId:131,isCFGNode:"True",location:"79:1:1637:1652",type:"ReturnStatement"})

(n147 {childNum:"12",code:"Cleanup :",functionId:131,isCFGNode:"True",location:"78:0:1627:1634",type:"Label"})

(n193 {childNum:"5",code:"except",functionId:131,isCFGNode:"True",location:"60:1:1291:1296",type:"Statement"})

(n183 {childNum:"0",code:"ntStatus = GetExceptionCode ( )",functionId:131,isCFGNode:"True",location:"63:2:1331:1360",type:"ExpressionStatement"})

(n192 {childNum:"6",code:"(",functionId:131,isCFGNode:"True",location:"60:7:1297:1297",type:"Statement"})

(n181 {childNum:"1",code:"goto Cleanup ;",functionId:131,isCFGNode:"True",location:"64:2:1364:1376",type:"GotoStatement"})