Code Monkey home page Code Monkey logo

opal-backup's Introduction

Overview

OPAL is an extensible library for analyzing and engineering Java bytecode. OPAL is completely written in Scala and leverages Scala's advanced language features to provide a new and previously unseen level of flexibility and ease of use. OPAL was designed from the ground up with extensibility, adaptability and scalability (memory and performance-wise) in mind. Many parts of OPAL are either already parallelized, provide the necessary infrastructure to implement highly concurrent analyses or are at least thread-safe.

Main Projects

OPAL consists of several projects:

  • Common (OPAL/common): Contains general useful functions, data-structures (e.g. TrieMaps) and graph algorithms (e.g., computing strongly connected components, computing dominator information, etc.) useful when analyzing (byte) code.

  • Static Analysis Infrastructure (OPAL/si): Contains a generic lattice-based framework for the implementation of modularized static analyses.

  • Bytecode Infrastructure (OPAL/bi): The necessary infrastructure for parsing Java 1.0 - Java 16 bytecode.

  • Bytecode Disassembler (OPAL/da): A Java Bytecode Disassembler that provides a one-to-one representation of the class file and which can be used to create readable HTML representations of Java class files.

  • Bytecode Creator (OPAL/bc): Most basic infrastructure to engineer Java bytecode.

  • Bytecode Representation (OPAL/br): OPAL's base representation of Java bytecode. Implements all functionality to do basic analyses of Java class files.

  • Abstract Interpretation Framework (OPAL/ai): Implementation of an abstract interpretation based framework that can be used to easily implement analyses at different levels of precision.

  • Three Address Code (OPAL/tac): Provides two 3-address code based intermediate representation. A naive one which is directly created based on the bytecode, and a higher-level SSA-like representation which directly provides a CFG as well as Def-Use information using the results of a basic abstract interpretation.

  • Dependencies Extraction (OPAL/de): Provides support for extracting and analyzing a project's source code dependencies. This project is the foundation for projects to, e.g., check architectures.

  • Architecture Validation (OPAL/av): A small framework to check a project's implemented architecture against a specified one.

  • Framework (OPAL/framework): Basically just aggregates all sub-projects to make it possible to easily get a consistent snapshot of all sub-projects. In general, it is recommended to declare a dependency on this project when you want to use OPAL.

  • Demos (OPAL/demos): Contains working code samples that demonstrate how to use OPAL. The code in the Demo project is primarily meant as a teaching resource. To start the examples, start the sbt console (Scala Build Tools) and change the current project to "Demos" (project Demos). After that you can run several small demo analyses.

  • Hermes (OPAL/TOOLS/hermes): A framework to run various code queries against sets of projects.

  • BugPicker (OPAL/TOOLS/bp): A tool to find control-/data-flow dependent issues in source code. The kind of issues that are identified range from useless defensive checks to bugs that lead to (unhandled) runtime exceptions.

Developer Tools

OPAL also comes with a growing number of tools that are intended to help developers to become familiar with Java Bytecode and/or OPAL. These projects are found in the folder DEVELOPING_OPAL/tools and can be run using the SBT console.

Building OPAL

The following applies to the "Master/Develop" branch.

OPAL uses SBT as its build tool and working with OPAL is particularly easy using the SBT console. Make sure that you have Java 8 at least update 171, Scala 2.12.13 and SBT 1.4.6 installed and running and that SBT can use at least 4GB of RAM (-Xmx4G). Download a recent snapshot of OPAL or clone the repository. Go to OPAL's root folder.

  • Call sbt cleanBuild. This compiles all core projects (including tests), generates the project-wide ScalaDoc documentation and publishes the project to your local ivy directory.
  • [Optional - but highly recommended] Edit the file local.sbt and specify the two system properties (JAVA_OPTS): -Dorg.opalj.threads.CPUBoundTasks=8 -Dorg.opalj.threads.IOBoundTasks=24 - set the values to appropriate values for your machine (CPUBoundTasks === "Number of real CPUs (Cores)", IOBoundTasks === "Number of (hyperthreaded) cores * 1 .5"). You can also set these properties when using sbt by typing:
    eval sys.props("org.opalj.threads.CPUBoundTasks") = "1".
  • Call sbt test to run the unit tests and to test that everything works as expected. Please note, that some tests generate some additional (colored) output. However, as long as all tests succeed without an error, everything is OK. If sbt test fails, it may be due to insufficient memory. In this case it is necessary to start SBT itself with more memory.
  • Call sbt it:test to run the integration test suite. Executing this test suite may take very long (on a fast desktop with 32GB and 8 Cores it takes ~2h).

You are ready to go.

Troubleshooting

When you encounter problems in building OPAL, please consider the following options.

  • Ensure that the correct file encoding is used by your editor/sbt/... All files use UTF-8 encoding. (This is in particular relevant when you are using Windows.)
  • Increase the heap size; to build and run all tests you should give sbt at least 12GB.

Using OPAL

To get started, go to the project webpage. Additionally, the code in the Demos project contains many short(er) examples that demonstrate how to solve commonly recurring tasks. Most examples can directly be executed.

Example Usage

Start the sbt console. (In OPAL's root folder call sbt on the command line.) Change the project to Demos using the command project Demos and type run to run one of the demos.

Further Information

opal-backup's People

Contributors

a10r avatar amuttsch avatar andrepacak avatar bhermann avatar blaisorblade avatar cystalcreation avatar danpingxu avatar delors avatar dsiebert avatar eichhrn avatar erich-wittenbeck avatar errt avatar fbussjor avatar israphim avatar larsschulte avatar lschulte avatar marceichler avatar mariotragesercofinpro avatar marius-d avatar mitschke avatar mjacobasch avatar molty avatar mreif avatar riadhchtara avatar rkolosovs avatar roteremil avatar simlei avatar steph0r avatar torunr avatar waelkh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

opal-backup's Issues

TACAI L2 Tests fail

On the current develop branch, two TACAI tests for the L2 domain fail. The failing tests are:

  • inlining "trivial" method calls and using origin based def/use information should enable more precise tracking of instances

  • chained method calles ("Builder Pattern")

Adding type parameters to PropertyComputationResult

Taken from Bitbucket#142:

In FPCF we can compute various properties for a single entity. The analyses do so by computing a PropertyComputationResult.
However, there is no type guarantee that the analyses always returns the intended result, e.g. an escape analysis should compute a Result that captures an escape property and not a purity property.
The scenario becomes more realistic when it comes to wrapper properties, e.g. representing aggregated results.
In cases where different entities/properties should be computed it would be still possible to use Entity/Property as type parameter

Stronger type guarantees would be still desirable, but are way more complex to archive in the current setting.

Readme.md for the design decisions for the call graph analyses

Taken from Bitbucket#175

During the development of the pluggable call graph framework, we made some design decisions, like direct vs. indirect callees, incomplete callsites, high soundness mode, various callees properties, the handling of the thread api (mark callers as vm reachable but callees as incomplete).
Framework users should have a starting point to get an overview about these decisions.

Assertion in DeclaredMethodsKey

Assertion Error in com.botbrew.basil_24-AppTrackerEncryptor.jar Exception in thread "main" org.opalj.concurrent.ConcurrentExceptions: concurrent exceptions occurred Suppressed: java.lang.AssertionError: assertion failed: creation of declared methods failed: DefinedMethod(declaringClassType=com.viewpagerindicator.IcsLinearLayout,definedMethod=com.viewpagerindicator.IcsLinearLayout{ private boolean hasDividerBeforeChildAt(int) }) vs.(new) DefinedMethod(declaringClassType=com.viewpagerindicator.IcsLinearLayout,definedMethod=android.widget.LinearLayout{ protected boolean hasDividerBeforeChildAt(int) })} at scala.Predef$.assert(Predef.scala:223) at org.opalj.br.analyses.DeclaredMethodsKey$.insertDeclaredMethod$1(DeclaredMethodsKey.scala:109) at org.opalj.br.analyses.DeclaredMethodsKey$.$anonfun$compute$14(DeclaredMethodsKey.scala:200) at org.opalj.br.analyses.DeclaredMethodsKey$.$anonfun$compute$14$adapted(DeclaredMethodsKey.scala:197) at org.opalj.collection.immutable.ConstArray.foreach(ConstArray.scala:40) at org.opalj.br.analyses.DeclaredMethodsKey$.$anonfun$compute$5(DeclaredMethodsKey.scala:197) at org.opalj.br.analyses.DeclaredMethodsKey$.$anonfun$compute$5$adapted(DeclaredMethodsKey.scala:114) at org.opalj.concurrent.package$.$anonfun$parForeachArrayElement$2(package.scala:223) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:658) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

See Bitbucket for files to reproduce.

Incompatible values on apply

There is a conceptual problem in our handling of incompatible values and suppression. While the PropertyStore ensures that no updates are propagated for incompatible results, the initial apply that retrieves the initial value may return an incompatible value.

I see two possible remedies for that, both require adapting all analyses:
Either we could change the PropertyStore interface to take an additional parameter to apply to specify whether the analysis can handle upper or lower bounds, both or only final results. If the current value does not match this, the apply method would return the corresponding EPK instead.
The other possible solution is for all analyses to expect an incompatible value from the apply invocations and just treat it the same way that an EPK is treated.

Note that currently this is not an issue in practice because we have no analyses that could produce incompatible values unexpectedly. If we were to add such analyses, the problem should become apparent through match not exhaustive exceptions or similar.

TACAI fails on OpenJDK 15

On OpenJDK 15, the PurityTests fail with a NullPointerException during TACAI, which didn't happen in OpenJDK 11.

Field mutability should identify external initialization

Taken from Bitbucket#156

Sometimes, effectively final fields are not initialized via a constructor, but via direct assignments on a newly created, not yet escaped object. This pattern is common e.g. for implementing clone:

int i;

public Foo clone(){
Foo f = new Foo();
f.i = i;
return f;
}

Identifying such patters would improve the precision of the field mutability analysis.

Wrong mapping of tac indices

openjdk 8 version 265
class: sun.nio.cs.IBM874
The clinit method is not correct map from bytecode to tac:

pc2Index-array:
19->7
20->0
21->8
22->0
23->0
24->8
25->0

bytecode:
16 95 SIPUSH(256)
19 95 NEWARRAY(char[])
21 95 PUTSTATIC(sun.nio.cs.IBM874.c2bIndex : char[])
24 98 GETSTATIC(sun.nio.cs.IBM874.b2c : char[])
27 98 ASTORE_0
28 99 ACONST_NULL
29 99 ASTORE_1
30 101 LDC("€�������������������������� �����่กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯะัาำิีึืฺุู้๊๋์฿เแโใไๅๆ็่้๊๋์ํ๎๏๐๑๒๓๔๕๖๗๘๙๚๛¢¬¦  �������\b\t\n� \r������������������ !"#$%&'()*+,-./0123456789:;<=>?@abcdefghijklmnopqrstuvwxyz[\]^_`abcdefghijklmnopqrstuvwxyz{|}~�")
32 101 INVOKEVIRTUAL(java.lang.String{ char[] toCharArray() })
35 101 ASTORE_0
36 102 ALOAD_0

tac:
6: lv6 = 256
7: lv7 = new char[{lv6}]
8: sun.nio.cs.IBM874.c2bIndex = {lv7}
9: lv9 = null
10: lva = "€�������������������������������่กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯะัาำิีึืฺุู้๊๋์฿เแโใไๅๆ็่้๊๋์ํ๎๏๐๑๒๓๔๕๖๗๘๙๚๛¢¬¦  �������\b\t\n � \r������������������ !"#$%&'()*+,-./0123456789:;<=>?@abcdefghijklmnopqrstuvwxyz[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~�"
11: lvb = {lva}/java.lang.String/.toCharArray()

Switch upper/lower bounds

Within FPCF, bottom is the "worst" value, or most imprecise value, and top is the best/most precise one.
E.g. for a call graph, bottom is the set of all edges, and top the empty set.
Thus, a lower bound would be the set of edges that can be included at worst, an the upper-bound the set that must be included according to the analysis.
In most related work, in particular in the field of abstract interpretation, this is vise-versa.

We should refactor the whole framework as well as all analyses, such that we use the terms common in literature.

Implement equals on TACode

Taken from Bitbucket#166

TACode currently only supports reference equality which leads to an exception in the property store in debug mode when no TACAI analysis is scheduled and the fallback is computed more than once.

Improving the purity analysis

Taken from Bitbucket#157

Can we do something about the following issue (from the ASE paper):
"In contrast to Benton and Fischer, we treat methods reading the receiver's mutable state as \property{side-effect free} instead of \property{externally pure}.
This enables us to classify methods with calls on non-confined receiver objects as \property{side-effect free} rather than \property{impure}.
The drawback is that a caller that invokes such methods on a confined receiver object can not be \property{pure} anymore."

FPCFAnalysesIntegrationTest.config is not maintainable

Taken from Bitbucket#174

The config file specifies the analyses to be run in the integration test. However, it seems to be insufficient to maintain on framework changes.
I'd suggest to just implement the integration test with a fixed set of analyses.

Findings are printed twice

From Bitbucket#180:

When we run the "Demos....FieldAndArrayUsageAnalysis" analysis for the rt.jar of the JDK 8 all findings are listed twice.

[info] sun.security.pkcs11.P11Key$P11RSAPrivateNonCRTKey{ private void fetchValues() }:
[info]   574 new DefinitionSite(m=sun.security.pkcs11.P11Key$P11RSAPrivateNonCRTKey{ private void fetchValues() }, pc=34
[info]   574 ArrayStore(pc=44,UVar(defSites={6},value=sun.security.pkcs11.wrapper.CK_ATTRIBUTE[]),UVar(defSites={12},value=AnIntegerValue),UVar(defSites={13},value=sun.security.pkcs11.wrapper.CK_ATTRIBUTE))}
[info]                                                             
[info] sun.security.pkcs11.P11Key$P11RSAPrivateNonCRTKey{ private void fetchValues() }:
[info]   574 new DefinitionSite(m=sun.security.pkcs11.P11Key$P11RSAPrivateNonCRTKey{ private void fetchValues() }, pc=34
[info]   574 ArrayStore(pc=44,UVar(defSites={6},value=sun.security.pkcs11.wrapper.CK_ATTRIBUTE[]),UVar(defSites={12},value=AnIntegerValue),UVar(defSites={13},value=sun.security.pkcs11.wrapper.CK_ATTRIBUTE))}
[info] 

Virtual Call Resolution does not work correctly for private methods

Taken from Bitbucket#179:

ProjectLike's virtualCall resolution does not work in the following code evolution scenario. The JVM runs this code (please see the attached files. B has a main).

Class Structure

class A {
   protected void method(){\* do  something*\}
}

class B extends A {
   private void method(){/* do something*/}

  public static void main(String ... args) {
    var B = new B();
    b.method(); // this is an INVOKEVIRTUAL
  }
}

Problem: b.method is resolved to A.method when an invokevirtual is used.

sbt console code leading to this error:

val file = new java.io.File(dir)
val project = org.opalj.br.analyses.Project(file)
val methods = project.allMethodsWithBody
val main = methods.filter(_.name =="main")
val body = main.body.get
val invoke = body.collectFirst{
   case x @ INVOKEVIRTUAL(org.opalj.br.ObjectType("B"),"m", _) => x
}.get
val results = project.virtualCall("", invoke)

OUTPUT: results: scala.collection.Set[org.opalj.br.Method] = Set(protected void m())

The respective classfiles are attached. "StrangeCallCreator.scala" is the DSL script that engineered these classes.

files.zip

May-alias analysis/domain for Fields and Arrays to improve the escape analysis

Taken from Bitbucket#176:

In the current escape analysis, more than 90% of all entities are flagged with AtMost(...) (which is currently equivalent to GlobalEscape), because they were stored into arrays or fields.

Consider the following example:

static Object global = null;
void foo() {
Object o = new Object(); // may be local, iff the array arr is local and no read from the array escapes
Object[] arr = new Object[] { o }; // the array is local
Object o2 = arr[0]; // here the three-address code provides no information that o2 might be an alias of o
// global = o2; // would let o escape. if the stmt is omitted, o would not escape.
}
As a first step we would need such a field/array may alias analysis. Afterwards we would need to modify the escape analysis.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.