rather than failing based on very simple criteria, allow more complex failure definitions to be created, and different ways to react to those fails, e.g.: terminate, inject long running query into results, etc.

Hi,
I'm trying to package ldbc_driver in order to import it as a jar and implement a new test driver for our SUT. When running mvn package I get the following response:
Failed tests: shouldLoadFromConfigFile(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadReadTest): (..)
shouldWriteToCsvWhileRunningWorkload(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadReadTest): (..)
shouldAssignMonotonicallyIncreasingScheduledStartTimesToOperations(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadTest): (..)
operationsShouldHaveMonotonicallyIncreasingScheduledStartTimesAfterSplitting(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadTest): (..)
shouldLoadFromConfigFile(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadTest): (..)
shouldWorkWhenOnlyWriteOperationsAreEnabled(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadTest): (..)
shouldWriteToCsvWhileRunningWorkload(com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkloadTest): (..)
shouldLoadFromConfigFile(com.ldbc.driver.workloads.simple.SimpleWorkloadTest): (..)
Thanks
Tomer

Instead of:

INFO  com.ldbc.driver.Client  Validation Result: PASS
Database: com.ldbc.snb.interactive.vendor.DatabaseConnectorImpl
  ***
  Successfully executed 14 operation types:
    7 of      LdbcQuery1
    7 of      LdbcQuery10
  ***
  Missing handler implementations for 0 operation types:
  ***
  Unable to execute 0 operations:
  ***
  Incorrect results for 0 operations:
  ***

Output:

INFO  com.ldbc.driver.Client  Validation Result: PASS
Database: com.ldbc.snb.interactive.vendor.DatabaseConnectorImpl
  ***
  Successfully executed 14 operation types:
    7/7 of      LdbcQuery1
    7/7 of      LdbcQuery10
  ***
  Missing handler implementations for 0 operation types:
  ***
  Unable to execute 0 operations:
  ***
  Incorrect results for 0 operations:
  ***

Work up
Reach steady state (or satisfy some predicate)
Start measurement
Run actual workload

details...

To the Workload Statistics output it would be very useful to know what the value of SafeT is, i.e., the min(op.scheduledStartTime - op.dependencyTime)
That information would be useful when tuning/developing workloads

Hi,
The guys from the other side of the C++ / Java divide are asking if you plan to port the driver code to C++ at any point? One person pointed out that TPC-C gives a detailed spec of the test driver and c++ reference code rather than a mandatory driver. I understand LDBC is using a different approach, but this may inhibit usage of C++ based systems efficiently as all query calls will need to go through JNI adapters which at large throughput may incur significant overhead.
Your thoughts?
Tomer

then output files would be something like: NAME-results.json, NAME-configuration.properties

When a delay to execute an operation on time occurs:

• calculate how long it was delayed
• calculate the operations that would have been executed in that time
  • based on the workload mix
  • based on interval between operations
  • could do this by creating a (small) sample of the full workload during driver initialization, creating a looping iterator over that sample, then injecting/logging-to-metrics "fake" operations from this iterator, to account for the "omitted" operations
    • may need multiple samples from different workload timeframes to account for workloads that change over time
    • based on which operation failed, choose appropriate sample from appropriate timeframe
    • need to estimate scheduledStartTime, actualStartTime (perhaps equal to scheduledStartTime or perhaps equal to mean from previous operations?), runDuration (taken from mean of previous operations, for example)
    • would means from previous operations include the times from the injected/fake operations?

Alternatively, when a delay to execute an operation on time occurs:

• rather than executing that operation, send it to a "missed operations" place
• do not execute it
• do not log the operation in any other way
  tolerate up to some amount of such operations
• writes would still have to be executed, but reads do not need to be
• upon completion of the benchmark assign a some start and finish time to all "missed operations", based on means/percentiles/whatever of other operations of the same type/that were executed at the same period/whatever. place these in "corrected operations"
• include metrics from "corrected operations" to the final report metrics

Could possibly do this from a Delay Policy implementation

If the metrics logging service has this looping iterator the "fake" operations would not need to all get sent over a queue to the logging thread. Instead, just 1 message, containing a parameter communicating "number of fake operations to create", would be sent to the logging thread and it would deal with the rest

Make configurable:

• terminate
• correct from sample
• something else?
• ...

file would be named BENCHMARK_RUN_NAME-configuration.properties

link to validation repository in ldbc-dev
add validation documentation to ldbc_driver wiki

Hi,
The dependency on neo4j snapshot repo is a nuisance for us (complicated, behind firewall + proxy + lots of annoying configuration hoops to jump through) corporate users. Any chance of sticking with released versions?
Thanks
Tomer

multiple driver processes should coordinate (maintain GCT, etc.) to run 1 benchmark in a more scalable manner

details...

consider making OperationHandlerExecutor implementations blocking on "execute()", so there is some back pressure going back to the ExecutorService implementations, otherwise this throughput mode will just fill the queue to the OperationHandlerExecutor

This is what happens when trying to run on SF1 generated by datagen 0.2:

INFO  com.ldbc.driver.Client  Loaded DB: hpl.alp2.titan.drivers.interactive.TitanFTMDb
INFO  com.ldbc.driver.Client  Scanning workload streams to calculate their limits...
ERROR com.ldbc.driver.Client  Client terminated unexpectedly
com.ldbc.driver.ClientException: Error loading workload class: com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkload
    at com.ldbc.driver.Client$ExecuteWorkloadMode.init(Client.java:251)
    at com.ldbc.driver.Client.<init>(Client.java:137)
    at com.ldbc.driver.Client.main(Client.java:49)
Caused by: java.lang.NumberFormatException: Invalid digit character -3 '�' where the original char was '-'
    at com.ldbc.driver.csv.Extractors.digit(Extractors.java:660)
    at com.ldbc.driver.csv.Extractors.extractLong(Extractors.java:652)
    at com.ldbc.driver.csv.Extractors.access$000(Extractors.java:63)
    at com.ldbc.driver.csv.Extractors$LongExtractor.extract(Extractors.java:242)
    at com.ldbc.driver.csv.BufferedCharSeeker.extract(BufferedCharSeeker.java:164)
    at com.ldbc.driver.workloads.ldbc.snb.interactive.WriteEventStreamReaderCharSeeker$EventDecoderAddPerson.decodeEvent(WriteEventStreamReaderCharSeeker.java:110)
    at com.ldbc.driver.workloads.ldbc.snb.interactive.WriteEventStreamReaderCharSeeker$EventDecoderAddPerson.decodeEvent(WriteEventStreamReaderCharSeeker.java:71)
    at com.ldbc.driver.generator.CsvEventStreamReaderTimedTypedCharSeeker.getNextEvent(CsvEventStreamReaderTimedTypedCharSeeker.java:81)
    at com.ldbc.driver.generator.CsvEventStreamReaderTimedTypedCharSeeker.next(CsvEventStreamReaderTimedTypedCharSeeker.java:42)
    at com.ldbc.driver.workloads.ldbc.snb.interactive.WriteEventStreamReaderCharSeeker.next(WriteEventStreamReaderCharSeeker.java:63)
    at com.ldbc.driver.workloads.ldbc.snb.interactive.WriteEventStreamReaderCharSeeker.next(WriteEventStreamReaderCharSeeker.java:16)
    at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1162)
    at com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkload.getStreams(LdbcSnbInteractiveWorkload.java:291)
    at com.ldbc.driver.Workload.streams(Workload.java:45)
    at com.ldbc.driver.WorkloadStreams.createNewWorkloadWithLimitedWorkloadStreams(WorkloadStreams.java:175)
    at com.ldbc.driver.WorkloadStreams.createNewWorkloadWithLimitedWorkloadStreams(WorkloadStreams.java:165)
    at com.ldbc.driver.Client$ExecuteWorkloadMode.init(Client.java:245)
    ... 2 more

it may be more performance to have an implementation of OperationHandlerExecutor (in addition to SingleThreadedOperationHandlerExecutor and ThreadPoolOperationHandlerExecutor) that does the execution of the operation handler in the same thread: SameThreadOperationHandlerExecutor.

Plus:

• When supporting multiple Synchronous streams in the same process this will reduce thread count
• There will be less overhead: put handler in queue, take handler from queue, execute VS execute

Minus:

• In some cases (when reducing thread count is not necessary, i.e., there are many spare cores) it may be slower as the submitting of work to the OperationHandlerExecutor and the actual executing will be in the same thread.

Need to benchmark

Write, Read, Modify, Read-Modify-Write, Scan (consider removing Scan from SimpleWorkload)

Hi,
It would be great to have instructions here: https://github.com/ldbc/ldbc_driver/wiki/Running-a-Benchmark
on how to run the client embedded in another Java program. i.e. maven dependency, MWE, etc.
I think it's reasonable that a company would want to run the benchmark from it's own test-director and collect the results.
Thanks
Tomer

add support for serializing a workload definition to file
persist configuration file at the same time, so others can use the same settings
add support for deserializing a workload from file

For SF300, at this stage it will hang more than 20minutes with this config:
Name: LDBC
DB: com.ldbc.driver.workloads.ldbc.snb.interactive.db.VirtuosoDb
Workload: com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkload
Operation Count: 1,000
Worker Threads: 1
Status Display Interval: 00:02.000.000 (m:s.ms.us)
Time Unit: MILLISECONDS
Results Directory: /home/sib300/driver/results
Create Results Log: false
Time Compression Ratio: 0.0001000
Peer IDs: []
Validation Creation Params: null
Database Validation File: null
Validate Workload: false
Calculate Workload Statistics: false
Spinner Sleep Duration: 00:00.000.000 (m:s.ms.us) / 0 (ms)
Print Help: false
Ignore Scheduled Start Times: false
User-defined parameters:
endpoint = localhost:1210
ldbc.snb.interactive.LdbcQuery10_enable = false
ldbc.snb.interactive.LdbcQuery10_freq = 329
ldbc.snb.interactive.LdbcQuery10_interleave = 329
ldbc.snb.interactive.LdbcQuery11_enable = false
ldbc.snb.interactive.LdbcQuery11_freq = 371
ldbc.snb.interactive.LdbcQuery11_interleave = 371
ldbc.snb.interactive.LdbcQuery12_enable = false
ldbc.snb.interactive.LdbcQuery12_freq = 407
ldbc.snb.interactive.LdbcQuery12_interleave = 407
ldbc.snb.interactive.LdbcQuery13_enable = false
ldbc.snb.interactive.LdbcQuery13_freq = 413
ldbc.snb.interactive.LdbcQuery13_interleave = 413
ldbc.snb.interactive.LdbcQuery14_enable = false
ldbc.snb.interactive.LdbcQuery14_freq = 748
ldbc.snb.interactive.LdbcQuery14_interleave = 748
ldbc.snb.interactive.LdbcQuery1_enable = false
ldbc.snb.interactive.LdbcQuery1_freq = 255
ldbc.snb.interactive.LdbcQuery1_interleave = 255
ldbc.snb.interactive.LdbcQuery2_enable = true
ldbc.snb.interactive.LdbcQuery2_freq = 500
ldbc.snb.interactive.LdbcQuery2_interleave = 500
ldbc.snb.interactive.LdbcQuery3_enable = false
ldbc.snb.interactive.LdbcQuery3_freq = 1189
ldbc.snb.interactive.LdbcQuery3_interleave = 1189
ldbc.snb.interactive.LdbcQuery4_enable = false
ldbc.snb.interactive.LdbcQuery4_freq = 421
ldbc.snb.interactive.LdbcQuery4_interleave = 421
ldbc.snb.interactive.LdbcQuery5_enable = false
ldbc.snb.interactive.LdbcQuery5_freq = 877
ldbc.snb.interactive.LdbcQuery5_interleave = 877
ldbc.snb.interactive.LdbcQuery6_enable = false
ldbc.snb.interactive.LdbcQuery6_freq = 3728
ldbc.snb.interactive.LdbcQuery6_interleave = 3728
ldbc.snb.interactive.LdbcQuery7_enable = false
ldbc.snb.interactive.LdbcQuery7_freq = 432
ldbc.snb.interactive.LdbcQuery7_interleave = 432
ldbc.snb.interactive.LdbcQuery8_enable = false
ldbc.snb.interactive.LdbcQuery8_freq = 34
ldbc.snb.interactive.LdbcQuery8_interleave = 34
ldbc.snb.interactive.LdbcQuery9_enable = false
ldbc.snb.interactive.LdbcQuery9_freq = 3709
ldbc.snb.interactive.LdbcQuery9_interleave = 3709
ldbc.snb.interactive.LdbcShortQuery1PersonProfile_enable = false
ldbc.snb.interactive.LdbcShortQuery2PersonPosts_enable = false
ldbc.snb.interactive.LdbcShortQuery3PersonFriends_enable = false
ldbc.snb.interactive.LdbcShortQuery4MessageContent_enable = false
ldbc.snb.interactive.LdbcShortQuery5MessageCreator_enable = false
ldbc.snb.interactive.LdbcShortQuery6MessageForum_enable = false
ldbc.snb.interactive.LdbcShortQuery7MessageReplies_enable = false
ldbc.snb.interactive.LdbcUpdate1AddPerson_enable = false
ldbc.snb.interactive.LdbcUpdate2AddPostLike_enable = false
ldbc.snb.interactive.LdbcUpdate3AddCommentLike_enable = false
ldbc.snb.interactive.LdbcUpdate4AddForum_enable = false
ldbc.snb.interactive.LdbcUpdate5AddForumMembership_enable = false
ldbc.snb.interactive.LdbcUpdate6AddPost_enable = false
ldbc.snb.interactive.LdbcUpdate7AddComment_enable = false
ldbc.snb.interactive.LdbcUpdate8AddFriendship_enable = false
ldbc.snb.interactive.data_dir = /2d6/mirkoOutputDir/300sf_csv
ldbc.snb.interactive.gct_delta_duration = 10000
ldbc.snb.interactive.max_write_event_start_time = 1359673209655
ldbc.snb.interactive.min_write_event_start_time = 1350748089001
ldbc.snb.interactive.num_events = 877373396
ldbc.snb.interactive.parameters_dir = /2d6/mirkoOutputDir/300sf_csv/substitution_parameters
ldbc.snb.interactive.update_interleave = 1
password = dba
printQueryNames = false
printQueryResults = false
printQueryStrings = false
queryDir = /2d1/ldbc/ldbc_snb_interactive_vendors/interactive/virtuoso/queries/sql
run_sql = true
user = dba

If I don't specify the updates dir, it will not happen.

inject new Operations into a workload based on the results of other queries, to generate more realistic workload scenarios

details...

how early should a handler be scheduled to compensate for context switching/dispatching latencies?
this needs to be tuned.
perhaps write a method that during initialization runs a test that measures how long it takes between dispatching an operation and having that operation actually start.
test should measure only the latency of dispatching, not the latency under high concurrency - which is just higher because there aren't enough compute resources to schedule the handler

Make an equivalent Workload definition to the YCSB CoreWorkload definition, but use Generators for everything, adding new generators where necessary

bounded blocking queues will prevent stream executors from fill up the stream and then trying to exit
bounded blocking queues will do a similar job to what the earlySpinner did, i.e., not submit jobs until it's nearly time to execute them, while removing the spinning/busy-wait and using less memory (for the queue)

for example parameters_dir --> ldbc.snb.interactive.parameters_dir

Hi,
I got the latest ldbc_driver this morning and now I get this error:

2146 [main] INFO  com.ldbc.driver.Client  - Validating database against expected results
 * Db: hpl.alp2.titan.drivers.interactive.TitanFTMDb
 * Validation Params File: C:\Users\sagit\workspace\ldbc_benchmarkFTM\titanFTM_driver\target\test-classes\workloads\validation_params.csv
2284 [main] ERROR com.ldbc.driver.Client  - Client terminated unexpectedly
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
at com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkload.marshalOperation(LdbcSnbInteractiveWorkload.java:1567)

I traced it to these lines:

    if (operationAsList.get(0).equals(LdbcQuery6.class.getName())) {
        long personId = ((Number) operationAsList.get(1)).longValue();
        String tagName = (String) operationAsList.get(2);
        int limit = ((Number) operationAsList.get(3)).intValue();
        return new LdbcQuery6(personId, tagName, limit);
    }

Thought maybe it was because I havent downloaded a new version of [https://github.com/ldbc-dev/ldbc_snb_interactive_validation] lately, so I did, and still got the same error.

Looks like there is an unexpected null in the LdbcQuery6 array (from first line of the validation_params.csv file):

["com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcQuery6",1099511740802,null,"John_Stuart_Mill",10]

Please look it up,
Thanks
Tomer

INFO should be cleanest and DEBUG most detailed

something like this...

INFO (for user feedback)

• message
  x no INFO prefix
  x no package and class name
  x no time
  ERROR (for developer/user feedback)
• message
• ERROR prefix
• package and class name
  x no time
  DEBUG (for developer feedback)
• message
• DEBUG prefix
• package and class name
• time

Hi,
Q2 validation is failing due to character map issues:
Expected Result: [..., LdbcQuery2Result{personId=2199023366815, personFirstName='Jos�', personLastName='Oliveira',...]
Actual Result : [..., LdbcQuery2Result{personId=2199023366815, personFirstName='José', personLastName='Oliveira',...]

Have you ran into this problem before?
Thanks
Tomer

does it make sense to have different ExecutionDelayPolicies for different components?

1. identify which components use these policies
2. investigate if it does make sense
3. assign different policies to the different components

Hi

When running the benchmark with all the queries enables (complex reads, short reads and updates), using the -stats options, short reads are not appearing in the summary. However, it seems that they are executed, as our implementation receives calls to short reads.

Arnau

add a format identifier to the top level of the results.json output file: "format_version": X, where X is an integer starting at 1

E.g.:

  {
    "format_version": 1,
    "unit": "MILLISECONDS",
    "start_time": 1397561979770,
    "finish_time": 1397561980670,

add a mode where Schedule Start Times are ignored, preferably with a simple flag in the configuration.
this will skip all busy waits that wait for scheduled start times and simply execute operations when a thread is free.

in addition to Operation.buildResult(...), would it be beneficial to have Operation.setFailed(...)?

at present only summary stats are collected:
means
percentiles
min, max
counts
is there a case for exhaustively gathering the unsummarized metrics for every operation, then performing analysis on this later?
seems useful, for example, to then plot changes over time in any slice desired

Integrate HdrHistogram https://github.com/giltene/HdrHistogram
Convert everything to nano seconds
Add tests to Metrics/Measurements layer

moving the check to the metrics service means it is only applied after execution and in a different thread, the benefits are two fold:

1. performance: the check will no longer occur in the execution hot path, and the metrics collection thread has little more to do than continually compute/update metrics in a separate thread. it will hopefully enable the driver to generate heavier workloads
2. context: as it is the metrics service that stores all... metrics about a benchmark run, it will simplify the task of having it (in the future) also keep track of how many operations were late, which were late, how to store late operations, etc. the spinner & handler will not have to be aware of this notion at all

add maximum tolerated query execution time check, possibly by query type as an extension.

Hi,
for some odd reason, the driver is complaining about a missing handler:

Missing handler implementations for 1 operation types: com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcQuery14

Even though I have them all:

 protected void onInit(Map<String, String> properties) throws DbException {
    String configFile = properties.get("confFile");
    URL u = this.getClass().getClassLoader().getResource(configFile);
    String confFilePath = u != null ? u.getPath() : null;
    if (confFilePath == null)
        throw new DbException("No db configuration file found for " + configFile);
    connectionState = new BasicDbConnectionState(confFilePath);
    registerOperationHandler(LdbcQuery1.class, LdbcQuery1Handler.class);
    registerOperationHandler(LdbcQuery2.class, LdbcQuery2Handler.class);
    registerOperationHandler(LdbcQuery3.class, LdbcQuery3Handler.class);
    registerOperationHandler(LdbcQuery4.class, LdbcQuery4Handler.class);
    registerOperationHandler(LdbcQuery5.class, LdbcQuery5Handler.class);
    registerOperationHandler(LdbcQuery6.class, LdbcQuery6Handler.class);
    registerOperationHandler(LdbcQuery7.class, LdbcQuery7Handler.class);
    registerOperationHandler(LdbcQuery8.class, LdbcQuery8Handler.class);
    registerOperationHandler(LdbcQuery9.class, LdbcQuery9Handler.class);
    registerOperationHandler(LdbcQuery10.class, LdbcQuery10Handler.class);
    registerOperationHandler(LdbcQuery11.class, LdbcQuery11Handler.class);
    registerOperationHandler(LdbcQuery12.class, LdbcQuery12Handler.class);
    registerOperationHandler(LdbcQuery13.class, LdbcQuery13Handler.class);
    registerOperationHandler(LdbcQuery14.class, LdbcQuery14Handler.class);
    registerOperationHandler(LdbcUpdate1AddPerson.class, LdbcQueryU1Handler.class);
    registerOperationHandler(LdbcUpdate2AddPostLike.class, LdbcQueryU2Handler.class);
    registerOperationHandler(LdbcUpdate3AddCommentLike.class, LdbcQueryU3Handler.class);
    registerOperationHandler(LdbcUpdate4AddForum.class, LdbcQueryU4Handler.class);
    registerOperationHandler(LdbcUpdate5AddForumMembership.class, LdbcQueryU5Handler.class);
    registerOperationHandler(LdbcUpdate6AddPost.class, LdbcQueryU6Handler.class);
    registerOperationHandler(LdbcUpdate7AddComment.class, LdbcQueryU7Handler.class);
    registerOperationHandler(LdbcUpdate8AddFriendship.class, LdbcQueryU8Handler.class);
}

Any ideas how to debug this?
Thanks
Tomer

Hi,
I'm trying to run my Db implementation and got a WorkLoad exception: parameters_dir
Traced back to the properties files which don't have this property and since it isn't documented [https://github.com/ldbc/ldbc_driver/wiki/Driver%20Configuration here]

I don't really know what to put there.
Thanks
Tomer

At present they return http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#currentTimeMillis()

This is inconsistent with the spec. We (LDBC) need to agree on a format and then implement it in the driver

does it make sense to have different error reporters for different components?
(1) identify which components use error reporters
(1) investigate if it does make sense
(2) assign different error reporters to the different components

Hi,
I can't find documentation for Operation Results in update queries. I thought I did it ok, but it turns out the validation code didn't check the update queries at all.
Guidance please ,
Thanks

Hi,
seems like putting the driver in validation mode causes it to ignore some parameters. Namely, query enablers and operation count. See the following output:

Parameters:
    Name:                           LDBC-SNB
    DB:                             hpl.alp2.titan.drivers.interactive.TitanFTMDb
    Workload:                       com.ldbc.driver.workloads.ldbc.snb.interactive.LdbcSnbInteractiveWorkload
    Operation Count:                10
    Worker Threads:                 4
    Status Display Interval:        00:20.000.000 (m:s.ms.us)
    Time Unit:                      MILLISECONDS
    Results Directory:              C:\Users\sagit\Desktop\results
    Create Results Log:             false
    Time Compression Ratio:         1.0000000
    Peer IDs:                       []
    Validation Creation Params:     null
    Database Validation File:       /C:/Users/sagit/workspace/ldbc_benchmarkFTM/titanFTM_driver/target/test-classes/workloads/validation_params.csv
    Validate Workload:              true
    Calculate Workload Statistics:  false
    Spinner Sleep Duration:         00:00.000.000 (m:s.ms.us) / 0 (ms)
    Print Help:                     false
    Ignore Scheduled Start Times:   false
    User-defined parameters:
        confFile = bdb_tmp.conf
        ldbc.snb.interactive.LdbcQuery10_enable = true
        ldbc.snb.interactive.LdbcQuery10_freq = 217
        ldbc.snb.interactive.LdbcQuery10_interleave = 10692458
        ldbc.snb.interactive.LdbcQuery11_enable = true
        ldbc.snb.interactive.LdbcQuery11_freq = 133
        ldbc.snb.interactive.LdbcQuery11_interleave = 6553442
        ldbc.snb.interactive.LdbcQuery12_enable = true
        ldbc.snb.interactive.LdbcQuery12_freq = 238
        ldbc.snb.interactive.LdbcQuery12_interleave = 11727212
        ldbc.snb.interactive.LdbcQuery13_enable = true
        ldbc.snb.interactive.LdbcQuery13_freq = 57
        ldbc.snb.interactive.LdbcQuery13_interleave = 2808618
        ldbc.snb.interactive.LdbcQuery14_enable = true
        ldbc.snb.interactive.LdbcQuery14_freq = 144
        ldbc.snb.interactive.LdbcQuery14_interleave = 7095456
        ldbc.snb.interactive.LdbcQuery1_enable = true
        ldbc.snb.interactive.LdbcQuery1_freq = 132
        ldbc.snb.interactive.LdbcQuery1_interleave = 6504168
        ldbc.snb.interactive.LdbcQuery2_enable = true
        ldbc.snb.interactive.LdbcQuery2_freq = 240
        ldbc.snb.interactive.LdbcQuery2_interleave = 11825760
        ldbc.snb.interactive.LdbcQuery3_enable = true
        ldbc.snb.interactive.LdbcQuery3_freq = 550
        ldbc.snb.interactive.LdbcQuery3_interleave = 27100700
        ldbc.snb.interactive.LdbcQuery4_enable = true
        ldbc.snb.interactive.LdbcQuery4_freq = 161
        ldbc.snb.interactive.LdbcQuery4_interleave = 7933114
        ldbc.snb.interactive.LdbcQuery5_enable = true
        ldbc.snb.interactive.LdbcQuery5_freq = 534
        ldbc.snb.interactive.LdbcQuery5_interleave = 26312316
        ldbc.snb.interactive.LdbcQuery6_enable = true
        ldbc.snb.interactive.LdbcQuery6_freq = 1615
        ldbc.snb.interactive.LdbcQuery6_interleave = 79577510
        ldbc.snb.interactive.LdbcQuery7_enable = true
        ldbc.snb.interactive.LdbcQuery7_freq = 144
        ldbc.snb.interactive.LdbcQuery7_interleave = 7095456
        ldbc.snb.interactive.LdbcQuery8_enable = true
        ldbc.snb.interactive.LdbcQuery8_freq = 13
        ldbc.snb.interactive.LdbcQuery8_interleave = 640562
        ldbc.snb.interactive.LdbcQuery9_enable = true
        ldbc.snb.interactive.LdbcQuery9_freq = 1425
        ldbc.snb.interactive.LdbcQuery9_interleave = 70215450
        ldbc.snb.interactive.LdbcShortQuery1PersonProfile_enable = true
        ldbc.snb.interactive.LdbcShortQuery2PersonPosts_enable = true
        ldbc.snb.interactive.LdbcShortQuery3PersonFriends_enable = true
        ldbc.snb.interactive.LdbcShortQuery4MessageContent_enable = true
        ldbc.snb.interactive.LdbcShortQuery5MessageCreator_enable = true
        ldbc.snb.interactive.LdbcShortQuery6MessageForum_enable = true
        ldbc.snb.interactive.LdbcShortQuery7MessageReplies_enable = true
        ldbc.snb.interactive.LdbcUpdate1AddPerson_enable = false
        ldbc.snb.interactive.LdbcUpdate2AddPostLike_enable = false
        ldbc.snb.interactive.LdbcUpdate3AddCommentLike_enable = false
        ldbc.snb.interactive.LdbcUpdate4AddForum_enable = false
        ldbc.snb.interactive.LdbcUpdate5AddForumMembership_enable = false
        ldbc.snb.interactive.LdbcUpdate6AddPost_enable = false
        ldbc.snb.interactive.LdbcUpdate7AddComment_enable = false
        ldbc.snb.interactive.LdbcUpdate8AddFriendship_enable = false
        ldbc.snb.interactive.parameters_dir = N:/SharedData/ALP2/LDBC/snb_validation
        ldbc.snb.interactive.short_read_dissipation = 0.2
        ldbc.snb.interactive.update_interleave = 49274

Client  Validating database against expected results
 * Db: hpl.alp2.titan.drivers.interactive.TitanFTMDb
 * Validation Params File: C:\Users\sagit\workspace\ldbc_benchmarkFTM\titanFTM_driver\target\test-classes\workloads\validation_params.csv
12:54:26.639 [main] DEBUG h.a.t.d.i.LdbcShortQuery1Handler - Short Query 1 called on person id: 5497558139109
12:54:26.719 [main] DEBUG h.a.t.d.i.LdbcShortQuery2Handler - Short Query 2 called on person id: 5497558139109
12:54:26.816 [main] DEBUG h.a.t.d.i.LdbcShortQuery3Handler - Short Query 3 called on person id: 5497558139109
12:54:27.070 [main] DEBUG h.a.t.d.i.LdbcShortQuery1Handler - Short Query 1 called on person id: 2199023256046
12:54:27.081 [main] DEBUG h.a.t.d.i.LdbcShortQuery2Handler - Short Query 2 called on person id: 2199023256046
12:54:27.933 [main] DEBUG h.a.t.d.i.LdbcShortQuery3Handler - Short Query 3 called on person id: 2199023256046
12:54:28.412 [main] DEBUG h.a.t.d.i.LdbcShortQuery1Handler - Short Query 1 called on person id: 81
12:54:28.419 [main] DEBUG h.a.t.d.i.LdbcShortQuery2Handler - Short Query 2 called on person id: 81
12:54:28.784 [main] DEBUG h.a.t.d.i.LdbcShortQuery3Handler - Short Query 3 called on person id: 81
12:54:28.834 [main] DEBUG h.a.t.d.i.LdbcShortQuery1Handler - Short Query 1 called on person id: 3298534883983
12:54:28.839 [main] DEBUG h.a.t.d.i.LdbcShortQuery2Handler - Short Query 2 called on person id: 3298534883983
12:54:28.962 [main] DEBUG h.a.t.d.i.LdbcShortQuery3Handler - Short Query 3 called on person id: 3298534883983
12:54:29.163 [main] DEBUG h.a.t.d.i.LdbcQueryU4Handler - U4 Adding forum 42949674103 , {title=Album 4 of Ge Wei, creationDate=1293862779830}
12:54:29.519 [main] DEBUG h.a.t.d.i.LdbcQueryU4Handler - U4 completed Adding forum 42949674103 

Notice how the operation count is 10 and all update queries are disabled and still I get more than 10 operations and an update query is called.
Did I do something wrong?
Thanks
Tomer

will allow many more concurrent queries to be simultaneously in flight

details...

I think LdbcUpdate5AddForumMembership.creationDate() should be LdbcUpdate5AddForumMembership.joinDate()
Thanks
Tomer

allow arbitrary ways of defining how a stream should be limited
could then implement GeneratorFactory.limit(Iterator, int) using that abstraction

Hi,
The update query numbers are mismatched. e.g. Q2 in the driver is addPostLike which is Q6 in the spec.
Thanks,
Tomer

make ConcurrentCompletionTimeService implementation configurable?
perhaps a method compares performance on target machine, using all different implementations, then chooses the one based on performance achieved

Add an integration test that tests the entire framework, top to bottom, to test if the defined operations, distributions, load are in fact generated by the framework

Things to test:

• Operation mix
• Generated load
• Distribution of each Operation parameter (keys and values)
• ...

find a way to support multiple synchronous executors in one process.

suggestion:
Workload returns a WorkloadStreams (class naming can change) instance that contains:

• Iterator<Operation<?>> partiallySynchronousStream()
• Iterator<Operation<?>> asynchronousStream()
• List<Iterator<Operation<?>>> synchronousStreams()

This would mean the "splitting" by execution mode no longer needs to be done by the WorkloadRunner, enabling:

• more lazy execution (splitting of streams can not be done in an efficient, lazy manner)
• multiple synchronous streams

The Workload implementation would need to take care of splitting, "manually"
The ExecutionMode enum could go