The C++ standard indicates that at least class-keys (that is, struct, class, union) should be semantically equivalent when forward declaring a type, however, MSVC++ uses a type's class-key as part of name-mangling.

It would be nice to explicitly add this requirement to our style guide, as this has bitten a few of us on the JIT before, and breaks usually result in hard to diagnose link errors.

[0] https://msdn.microsoft.com/en-us/library/695x5bes.aspx
[1] https://alfps.wordpress.com/2010/06/22/cppx-is-c4099-really-a-sillywarning-disabling-msvc-sillywarnings/
[2] http://stackoverflow.com/questions/4866425/mixing-class-and-struct

I've opened pull request #28 to disable the numa tests in the travis.ci builds as they appear to fail intermittently.

The tests need to be investigated to find out why they are failing.

There are some race conditions and other potentially incorrect assumptions in the OSX implementation of omrintrospect.

suspendAllPreemptive()
-- Does this function assume that task_threads() will always return >= the previous number of threads? What prevents a thread from exiting between the calls to task_threads() and thread_suspend()? Can you suspend a thread that has exited? Does task_threads() always return threads in the same order? Can you suspend a thread twice?
-- If this function encounters an error, then the runtime is stuck in a permanently bad state, correct?

I believe -DIPv6_FUNCTION_SUPPORT was related to some port library socket functions that were removed from OMR before the initial source code release. This flag should no longer be needed, and should be removed from all makefiles.

issueWriteBarrier() looks like a no-op:

(gdb) x/i 0x763e2b08
0x763e2b08 <issueWriteBarrier()>: bx lr

Follow-up to issue #51 (Merge generational GC (gencon) into OMR). Gencon has been partially integrated with gctest but was left in a disabled state with pull request #65. This fix will complete the task and enable gencon testing in fvtest.

It seems we have a couple places where we use the J9 version of OMR buildflag macros.
i.e.: J9VM_GC_CONCURRENT_SWEEP instead of OMR_GC_CONCURRENT_SWEEP

Note that the macro J9VM_GC_FINALIZATION does not currently have an OMR version, and a separate issue will be opened for it.

Add three new test configuration files to the OMR GC performance test runs in order to collect more performance data.

Would you like to add the configuration script "AX_PTHREAD" to your build specification?

• A process may hang if it uses omrcuda_* functions and terminates early by calling omrexit_shutdown_and_exit().
• Some device functions incorrectly record the presence of an initialized CUDA device context. If the first omrcuda function called by a thread is one of those below, the context will be incorrectly marked as valid, leading other functions to fail (e.g. omrcuda_linkerCreate).
  • omrcuda_deviceCanAccessPeer
  • omrcuda_deviceDisablePeerAccess
  • omrcuda_deviceEnablePeerAccess
  • omrcuda_deviceGetAttribute

We intend to use libdwarf in future ddr-related tooling.

libdwarf is packaged and installed differently for various Linux distros (Ubuntu, RedHat, SuSe). Add logic in the configure scripts to detect its location and format.

These changes will incorporate gencon technology into OMR. Current status (pre-merge) is code compiles cleanly and does not break any existing tests, gencon is enabled for example (fvtest) code and executes when enabled but there are issues to be dealt with before example (fvtest) is complete. These issues will be dealt with post-merge since getting this body of source code inline with OMR development of GetHub is primary concern.

The example (fvtest) code in omr/example/glue includes all of the integration points necessary to run gencon. This mainly involves changes to ObjectModel and the implementation of object scanners (MixedObjectScanner) and root scanners (ScavengerRootScanner, ScavengerBackOutScanner) for the fvtest "language", as well as changes to StartupManagerTestExample to accept "gcpolicy:gencon" as command-line option and enable gencon accordingly and to StartupManagerImpl to instantiate ConfigurationalGenerational (which in turn instantiates Scavenger). Other changes will be required post-merge, as noted above.

Object model and other changes have not at this point been reflected into omr/glue. This work relates to OMR documentation and will be addressed in a new issue post-merge.

If a native thread relies on thread-local storage destructors to detach from the omr thread library when calling pthread_exit they may not be able to successfully detach and leak memory.

Here is some pseudo code to demonstrate the issue:

main() {
    ....
    pthread_key_create(&key, detachThread);
    ....
    pthread_create(&thread, NULL, threadproc, (void *) omrVM);
    pthread_join(thread, NULL);
    pthread_key_delete(key);
    ....
}
static void *
threadproc(void *p) {
    omrthread_t omrThread = NULL;
    .....
    /* Attach to omr so you can use the omr thread library */
    omrthread_attach(&self);
    /* set key to some user data you need in the destructor */
    pthread_setspecific(key, (void *)myData);
    ....
    pthread_exit(NULL);
    .....
}
static void
detachThread(void *p)
{
    ....
    /* do something with my data */
    ....
    /* omrthread_self relies on a tls key which is now NULL */
    self = omrthread_self();
    if (NULL != self) {
        omrthread_detach(self);
    } else {
        /* leak memory */
    }
    ....
}

We are not properly shutting down the current global collector during calls to our public API OMR_GC_ShutdownCollector. This notably affects ConcurrentGC, which will leave behind a concurrent background helper thread.

I find that include guards like "omr_h" and "pool_api_h" are too short for the safe reuse of your header files (when they belong to an application programming interface).

If I am designing a language with a runtime that compiles to native code (e.g. Go, OCaml), is OMR still an appropriate choice? Or do the pieces (GC, threading) only work in the context of a VM?

Library looks great by the way, patiently awaiting more detailed documentation :)

Currently the style guidelines contain:

All objects and structures should be passed by address (i.e. *) rather than reference or value.
Non-object parameters that are not updated should be passed by value. C++ reference (i.e. &) arguments must not be used.
Rationale: Reference arguments can be problematic. For example, it's impossible to pass NULL for a reference object. They're just syntactic sugar to hide pointers, so we prefer to be explicit about how the argument is being passed.

C++ References enforce the equivalent of a non-NULL const pointer which is a stronger contract than merely declaring the pointer const. They help disambiguate the meaning of const and volatile in declarations. Furthermore, passing parameters by const reference allows for pass-by-reference optimizations without preventing the use of temporary r-values.

Generally, modern C++ strongly attempts to eschew pointers as much as is reasonably possible, passing by value or const reference whenever it can, passing by reference when needed. Pointers are found to obfuscate ownership and resource management responsibilities. The community has even gone so far as to produce a reference_wrapper class to be able to maintain (as far as possible) the conceptual integrity and non-nullable nature of references while being able to use them in container classes. For an example of this sentiment, see the 'no raw pointers' goal in well-received talk by Sean Parent on 'C++ seasoning'.

That being said, I will be the first to admit that passing non-const references for status / signal / reporting 'out' parameters can significantly obfuscate determining the intent of a function call in the context of a caller's code.

What pain points are we trying to address with this style guideline? Can we accomplish that without a universal prohibition on references?

I have a branch in my SOMpp fork using OMR and when I gather verbose:gc data OMR_VMThread names are not showing up where they are supposed to.

The lastname field in the response-info stanza and the threadName field in the largest-consumer stanza are printed as "OMR_VMThread [".

Migrate MasterGCThread code to OMR, so that other collectors needing concurrent operation (with explicit master GC thread) may make use of it.

Introduce core components for Concurrent Scavenger.

Currently our style guide requires that we use leading underscores to identify members in C++ classes. However there seems to be a community consensus that this isn't such a good idea because:

1. Identifiers containing a leading underscore are reserved implementation use in the global namespace.
2. Identifiers containing a leading underscore followed by a capital letter are universally reserved.
3. Identifiers containing two leading underscores are universally reserved.

Furthermore from the C standard:

All identifiers that begin with an underscore are always reserved for use as identifiers with file scope in both the ordinary and tag name spaces.

Frequent alternatives seem to be to either trail the identifier with an underscore member_ or to use more than just an underscore in the prefix m_member

[0] http://stackoverflow.com/questions/228783/what-are-the-rules-about-using-an-underscore-in-a-c-identifier

While completing changes to consume OMR in SOM++ I found an issue with the OMR_Initialize_VM API and with the default implementation of CollectorLanguageInterfaceImpl found in omr/glue and the example in omr/example/glue.

Both issues are related to multiple threads being used (either for VM helper threads or multi threaded languages). I will work on this issue.

J9STR_CODE_PLATFORM indicates that the string encoding is the "platform" encoding.
There are cases where OMR converts the result of system calls, e.g. getenv, to another encoding,
e.g. EBCDIOC->Latin-1 on z/OS and Wide->UTF-8 on Windows.
This change splits J9STR_CODE_PLATFORM into two constants:
J9STR_CODE_PLATFORM_RAW: the true encoding used by the operating system in the current locale
J9STR_CODE_PLATFORM_OMR_INTERNAL: the encoding taking into account implicit conversions by OMR

libdwarf lib and header install location varies by distro. For p & z-linux, binary packages aren't provided, and the consumer may need to build & install libdwarf in a custom location.

Add configure variables to enable ddrgen to be built with libdwarf from a custom location.

• Reference the coding standard doc
• Recommend referencing the issue number in the pull request
• Include the issue number in the commit comments (exact format is still TBD pending #46)

In a rare scenario of Scavenger backout, make sure that all parallel threads return same backout flag value from a scan cycle (being the main scan cycle or some of other smaller ones that may occur in miscellaneous clearing-scan phases toward the end of Scavenge cycle).

This is ensured by:

• the thread that first encounters backout condition will store index of the scan cycle
• threads will return value 'true' for the backout flag, only if their scan cycle index matches the one recorded by the first thread (thus avoiding a scenario where a late thread sees a backout flag of a fast thread in the next scan cycle)

Trivially a thread sync/barrier point would be put in between two scan cycles, but it would not be justified just for ensuring integrity of this flag, since this is a rare scenario.

I would like to point out that identifiers like "__ccsid_t" and "m__exitlock" do eventually not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

Currently the configure script downloads the configure files config.sub and config.guess. If the scripts can not download these files you can not compile the source.

I am suggesting that we check these files into the source to improve consumability of the Eclipse OMR project.

Fix needed for the following files:
port/linux386/omrsignal_context.c
port/linuxarm/omrsignal_context.c

Would you like to replace any double quotes by angle brackets around file names for include statements?

After I build the standalone OMR I ran "git status" and it showed lots of new files. We missed the .gitignore in the initial commit.

Would you like to add more error handling for return values from functions like the following?

• pthread_mutex_init ⇒ iconv_init
• strdup ⇒ ArgParser::parseOptions

A new API is being added to the port library that returns the number of files opened.

Stop using the -MD compile option, and stop using the generated *.d dependency files.

They don't significantly improve compile time.

Whenever there is significant file refactoring, some *.d files are no longer removed by "make clean", and the next attempt to compile usually fails due to incorrect dependencies specified by the obsolete *.d files.

After building and testing the OMR project with the default glue, there are a large number of untracked files that should likely be added to .gitignore.

Steps to reproduce:
< clone repo to local disk >

make -f run_configure.mk SPEC=osx OMRGLUE=./example/glue
make
make test
git status

git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
(use "git add ..." to include in what will be committed)

VerboseGC_59109_1457627456535.xml.001
VerboseGC_59109_1457627456535.xml.002
VerboseGC_59109_1457627456535.xml.003
VerboseGC_59109_1457627456758.xml
VerboseGC_59512_1457627886676.xml.001
VerboseGC_59512_1457627886676.xml.002
VerboseGC_59512_1457627886676.xml.003
VerboseGC_59512_1457627886843.xml
VerboseGC_64521_1457628095035.xml.001
VerboseGC_64521_1457628095035.xml.002
VerboseGC_64521_1457628095035.xml.003
VerboseGC_64521_1457628095225.xml
config.guess
config.sub
libbindthreadagent.dylib.dbg
libcpuLoadAgent.dylib.dbg
libinvalidAgentMissingOnLoad.dylib.dbg
libinvalidAgentMissingOnUnload.dylib.dbg
libinvalidAgentReturnError.dylib.dbg
libmemorycategoriesagent.dylib.dbg
libomrsig.dylib.dbg
libsampleSubscriber.dylib.dbg
libsltestlib.dylib.dbg
libsubscriberAgent.dylib.dbg
libsubscriberAgentWithJ9thread.dylib.dbg
libtraceNotStartedAgent.dylib.dbg
libtraceOptionAgent.dylib.dbg
libtraceagent.dylib.dbg
mmapTest1.tst
mmapTest10.tst
mmapTest11.tst
mmapTest13.tst
mmapTest2.tst
mmapTest3.tst
mmapTest4.tst
mmapTest5.tst
mmapTest6.tst
mmapTest7.tst
mmapTest8.tst
mmapTest9.tst
omrfile_test20_child_1_got_lock
omrfile_test20_child_2_done
omrfile_test20_file
omrfile_test22_child_1_got_lock
omrfile_test22_child_2_done
omrfile_test22_child_2_trying_for_lock
omrfile_test22_file
omrfile_test24_child_1_got_lock
omrfile_test24_child_2_done
omrfile_test24_child_2_trying_for_lock
omrfile_test24_file
omrfile_test25_child_1_got_lock
omrfile_test25_child_2_done
omrfile_test25_child_2_trying_for_lock
omrfile_test25_file
omrmmap_test5_child_1_checked_file
omrmmap_test5_child_1_mapped_file
omrmmap_test5_child_2_changed_file
traceLogTest.trc

Currently we reserved "maximum"(for maximum nursery space) native memory for the scan copy cache headers during vm startup, it might be requiring too much resource to initialize small heap usage application, so we will update CopyScanCacheList implementation to increase the size incrementally as the required or heap expansion. in order to balance between allocation size and allocation frequency. the new implementation are following the below rules.

1, the size of CopyScanCacheChunks are the same(includes initial size).
2, the size of Chunks(cache counts) =
if Xmnx <= 32MB ---> Xmnx / scavengerScanCacheMinimumSize
if 32MB < Xmnx < 4GB ---> MAX( Xmnx/16, 32MB) / scavengerScanCacheMinimumSize
if Xmnx >= 4GB ---> 256MB / scavengerScanCacheMinimumSize

3, when run out of CopyScanCache heads during Scavenger, try to allocate new chunk and append cache entries, if allocation is failed, allocate it in Heap
4, when heap expansion, if require cache entries for current new space is bigger than current count of cache entries, try to allocate new chunk and append cache entries
5, update cache overflow warning in verbose gc to monitor the performance of cache chunk allocation during scavenger.

...to the point that Travis-CI logs don't want to show it or anything after it, and I get tired of scrolling through it. It would be nicer if the test could report possibly a short summary along with pass/fail and save this output to a file that would be automatically cleaned up unless a failure is detected or an override command line option is provided.

Apache does not mandate the presence of a NOTICE file but recommend adding one: http://www.apache.org/legal/src-headers.html#notice

Example of a NOTICE file: https://www.apache.org/licenses/example-NOTICE.txt

Recursive VM exclusive access count is not properly maintained example code.

Pull request #20 added a .travis.yml file for the Travis.CI build engine. To enable continuous builds, the the github repo needs to enable travis integration. See https://docs.travis-ci.com/user/getting-started/

Note: You can only enable Travis CI builds for repositories you have admin access to.

Can an admins enable the travis integration?

DDR is a tool for analyzing core dumps. DDRGen is a utility that generates metadata for the DDR tool. In particular, it extracts information about data structures and macros from the runtime build. It outputs two artifacts that are consumed by the DDR tool chain (not included in this contribution):

1. superset - Metadata for generating language-specific accessor stubs for coredump data, which allow implementation of advanced debug functions in DDR.
2. blob - Metadata about structures, field offsets, and #defined values, used to generate the accessors.

The initial contribution of DDRGen covers:

• Extraction of debug data from DWARF debug data in output from GCC.
• Build and test on Linux/GCC x86-64 and -32.
  • Linux/GCC on p- and z- should "just work" after #88 is resolved.

DDRGen is disabled by default. Steps to build and run it:

make -f run_configure.mk SPEC=linux_x86-64 OMRGLUE=example/glue EXTRA_CONFIGURE_ARGS='--enable-DDR'

# Compiles ddrgen tools, but does no ddrgen processing
make

# Build ddr artifacts for sample test program
# vars can be customized to process full omr or runtime source tree
make -f ddr_artifacts.mk TOP_SRCDIR=tools/ddrgen/test DBG_FILE_LIST=tools/ddrgen/test/file_list

# Read the contents of the binary blob
./blob_reader blob.dat > blob.dat.strings

This is prep work for bringing the ddrgen tool into the OMR source tree.

Add configure option "--enable-DDR" to enable building the ddrgen tool.
Add top-level makefile targets for building the tool.

Infrastructure to support building DDR artifacts for the OMR code base.

• ddr_artifacts.mk to support building DDR artifacts over a source tree. This is separate from the primary makefile that compiles the tool because the consumer will probably want to customize the artifact build to scan their language source tree as well.
• Add "make ddrgen" target and cascading make targets to run preprocessor over the source tree.
• Add per-module make rules for preprocessing @ ddr annotated code.
• Add cleaning rules for generated .i artifacts.
• Don't run preprocessor on tools, tests, and third party code.

We should really use the proper name of the project in our README!!

Already submitted a pull request #31

The OSX version of the port library cannot map memory below 4GB.

OSX cannot map memory below 4GB, so omrmem_allocate_memory32() will always return NULL.
If we can't allocate memory below 4G, this may prevent us from supporting features like compressedrefs GC.

There may be an option that allows forcing mmap() below 4GB to succeed. It requires linking the executable (does not work for libraries) with option "-pagezero_size x".

For us to maintain good release changelogs, and be able to identify the release where an Issue gets fixed, we must be able to easily find (1) the Issue that corresponds to a commit; (2) all the commits needed to fix an Issue.

I propose updating the contribution guidelines to include:

• Every pull request must be associated with an Issue.
• Every commit comment must clearly reference an Issue.
• Provide a template for commit comments that makes it easy for us to find the issues in the git log.

After running "./configure OMRGLUE=./example/glue" as shown in the README, compilation fails:

make[2]: Entering directory `/home/angelal/git/omr.github/omr_glue_static_lib'
g++   -I. -I.././example/glue -I../include_core -I../omr/startup -I../gc/include -I../gc/startup -I../gc/base -I../gc/base/standard -I../gc/verbose -I../gc/verbose/handler_standard -I../gc/stats -I../gc/structs  -DUT_DIRECT_TRACE_REGISTRATION -I../include_core -I../nls -DLINUX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DIPv6_FUNCTION_SUPPORT -DJ9HAMMER -c  -MMD -fno-exceptions -fno-rtti -fno-threadsafe-statics -fPIC -ggdb -m64 -Wreturn-type -Werror -Wall -Wno-non-virtual-dtor -O3 -fno-strict-aliasing   -o .././example/glue/CollectorLanguageInterfaceImpl.o .././example/glue/CollectorLanguageInterfaceImpl.cpp
.././example/glue/CollectorLanguageInterfaceImpl.cpp:37:28: fatal error: omrExampleVM.hpp: No such file or directory
 #include "omrExampleVM.hpp"
                            ^
compilation terminated.

To make this work, you need to explicitly add --enable-OMR_EXAMPLE to the configure command.

From a certain point of view, it makes sense that --enable-OMR_EXAMPLE is required because I am compiling the example glue. But this is not very user-friendly.

One possible resolution is to default OMRGLUE to example/glue and enable_OMR_EXAMPLE to yes, so that no configure options are required for the trivial standalone case.

Another option is to make --enable_OMR_EXAMPLE set OMRGLUE to example/glue, so that the minimal standalone configuration is ./configure --enable-OMR_EXAMPLE.