Hi,

I may have missed something, but I'm wondering if there's support for object lookup using secondary indexes, as documented at http://docs.basho.com/riak/latest/tutorials/querying/Secondary-Indexes/, e.g.

curl localhost:8098/buckets/users/index/twitter_bin/jsmith123

Thanks,
Felix

Hi again,

As I'm working with the sync & async clients, I'm wondering if you guys have any examples of the async/callback pattern. I've worked with this type of pattern in languages like node.js before, but am still trying to understand best practices in C#. Using .NET 4.5 here FYI, so most of the goodies that are needed are already available.

Here's the example that I'm thinking of.

public int ReturnsAnInt() {
    RiakResult<RiakObject> result = Client.Get(someBucket, someKey);
    return ComputeIntFromRiakResult(result);
}

This is all well and good ... I end up getting my int after the Get completes, and I can go about my business. But it would be great if I could leverage the Async client in Riak to do this. I have a feeling that's quite possible, and that I'm probably just being a bit thickheaded about it....

public int WouldLoveToReturnAnInt() {
   // returns void
   Client.Async.Get(someBucket, someKey, delegate(RiakResult<RiakObject> result) { 
       int i = ComputeIntFromRiakResult(result);
    });

   // can't compile this -- 
   // is there some way that I can have my computed int back before I return in the top-level method?
}

Apologies for such a basic question here, and no rush to respond. But I think that once this is documented in CI it should help out other folks as well.

Cheers,
Felix

After removing OJ's ass from his shoulders and putting his head back on, he alleges to have removed the mess that was the CommandRunner. Deep down OJ always knew that the command approach for this problem was the wrong one, but for some reason his desire to have connection idling in there was more important.

Now that the CommandRunner is gone, and the extra thread and context switching along with it, the live unit tests take, on average, less than 1/3 of the time they used to take. That isn't a coincidence, and I don't think that connection idling is worth that kind of overhead. (OJ loves threading, someone let me write threading code.)

So it's been ripped out, and replaced with nothing, which means that the connection idling functionality no longer exists.

Adding idling isn't as simple as it sounds, given the pooling features that we have in place. Contention on the connection is a problem when you want to move it into an idle state (and hence kill off the connection) right at the same time as a client wants to use the connection. The simple option is to lock the connection each time it's used. Locking adds more overhead, and I'm really not keen on the idea of littering the connection code with locks.

There are a few ideas on how to manage this but OJ isn't sure how solid they are. In short, there's a big smelly dream, but that's it.

If I pull CorrugatedIron from nuget, my code bombs when I do Put or Get like so:

Could not load file or assembly 'Newtonsoft.Json, Version=4.0.2.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

If I build CorrugatedIron from source and then reference the freshly-built lib from my code everything works fine. It appears CorrugatedIron is tied to Newtonsoft 4.0.2.0 but nuget (being the eager beaver) pulls a later version 4.0.5.0.

Background

• Using CorrugatedIron / Riak 1.2 on a new project.
• Developing on Windows, with local Ubuntu VMs hosting Riak
• Using pretty standard fare repository interfaces along with dependency injection.
• As we go, we've ended up writing a number of in-memory variants of these repo interfaces to speed up unit tests, provide simulated data, etc, etc. Basically for any situation where we're testing something business-y and not persistence-y.

But I'm starting to think that there's a lot of ceremony that goes along with building these in-memory repos, and that it would be nice to push this responsibility off to CorrugatedIron.

Since the Riak datamodel is so simple, it can be pretty closely simulated with a ConcurrentDictionary<string, object>.

I haven't looked too closely at your code in a while, but I think that this would involve

• Implementing some new classes on top of the standard client interfaces
• Providing a config switch that denotes you're running on a fake Riak.
• Some combination of the two

I think the RavenDB client has a feature like that if I recall correctly.

Thoughts?

The entire lib needs to be profiled and benchmarked then compared against other clients to see how it performs. Bottlenecks and major issues with performance need to be highlighted and subsequently addressed.

Add vnode vclock flag in configuration to use the Riak 1.0 behavior of generating the vclock at the coordinating vnode

The SetObject method will bomb if users are attempting to save objects with a content type that is not Application/JSON, protocol buffers, or XML. text based content types should be allowed.

RiakConstants sets the default value for R to 2. This should be 4294967295-4 or 'default'.

Nagle's Algorithm:

In most cases — especially when using the PBC API, which tends to use small messages — clients should set the TCP_NODELAY flag on opened socket connections to Riak, which disables Nagle's Algorithm. Latency profiles having a minimum of 40 milliseconds often indicate the presence of Nagle on either end of the connection. If the client application has significant round-trip latency to the Riak cluster, disabling Nagle will have little effect, but for well-connected clients it can significantly reduce latency.

This would be getting the results of riak_admin status through code.

The DeleteBucket call currently only takes a RW value, and we should expose another overload which takes in the RiakDeleteOptions instance instead so that other values can be set, such as DW.

Verify that PUTs are being sent to the server with vector clocks properly attached.

If the vector clock is not set, we should examine the object being sent and pull the vclock from that. Should that not be an option, we throw an exception since lord only knows what's going to happen if we're not sending vclocks to the server. Some kind of sibling clown car, no doubt.

Remove Either<uint?, string> for R, RW, W, DW, and PR options. These options only take a uint32. Instead developers should make use of RiakConstants.QuorumOptions to define One, Quorum, All, or Default.

Hello

First of all, thanks for making CorrugatedIron available to everyone.

I was trying to run a long running M/R and realized I needed to set a higher timeout for the query.

AFAIK CorrugatedIron does not have any option to set the timeout for M/R queries am I right? If so, may I contribute to this project by implementing the option to set the query timeout?

Thanks in advance.

Remove calls to MarkClean(). This belongs in higher level code than the current version of CorrugatedIron.

The Apache License isn't present in all source code files (e.g. IRiakClient.cs). Add it.

Reduce the lines of code to configure and load an instance of a cluster for use in tools like a REPL.

Create scripts to automagically package up code for NuGet, binaries for github.

The key filter API doesn't fit with the rest of the API. Modify it and make it more fluent-y.

I have read this (and your announcement about intending to keep Xml), so am I familiar with your stance ;0
http://news.ycombinator.com/item?id=2799823

We are using your client internally against our test Riak cluster... and the config was one thing that immediately jumped out at me as something I didn't like. Having done a lot of ASP.NET / MVC / WCF development, I have a very strong bias against XML config at all costs -- it tends to grow out of hand and become a real beast to read, understand and maintain over time.

I would advocate for supporting both .config file based and code-based config, and based on your code, you can do this w/out a lot of extra effort. You're already most of the way there given your config is expressed through interfaces already. We're working around this by implementing the config interfaces ourselves with POCOs that have no connection to the .NET config classes, but I'd like to see you support that out of the box. All you need to provide really are hand-rolled validation routines that match up with the config validation attributes. (I've done this sort of thing internally on other projects)

I would also like to make a couple of comments about your remarks on the ycombinator thread.

• If you're going to setup a Riak cluster, which has to run on some flavor of nix, I don't see why anyone would *not setup a load balancer like HAProxy. I am decidedly not a Debian expert, but it took me about 20 mins to setup HAproxy to load balance the cluster. My point is, presumably any prod cluster will be setup by ops guys (or smart devs!) and setting up HAproxy is not hard... it would be one thing if Riak could run on Windows, but since it can't... I don't understand the stance there.
• By implementing load balancing in your client, you're more or less making the assumption that its the only one communicating with the cluster. If there are other client stacks talking to the cluster, then the accuracy of your load balancing is diminished. IMHO, it really shouldn't be the responsibility of the client toolkit to determine a load balancing strategy given it has no idea of what other clients might be doing.
• Recommendation for using protobufs is to use the 'least connected' algorithm since the protobuf connection is long running, which HAProxy and other proxies support out of the box. Round robin isn't the best choice there. To my previous point, this would be very difficult to implement correctly in CorrugatedIron if there are other clients communicating with Riak.

There's a good chance that we'll be building out a few pieces on node.js for instance. I think if you're a team looking at Riak, chances are you'll probably have at least a couple of client stacks working off it, which would again emphasize the need for a centralized load balancer.

So in short, I think it's awesome that your client does round robin load balancing -- and there are some scenarios where that might be useful. However, I think in any real production use, I can't see a lot of people using that. If I were to guess, I would expect that the majority of .NET teams would be deploying on Mono, where installing HAProxy is something that's more familiar anyhow. In our world, we'll be building out a service layer with ServiceStack, and almost certainly be deploying on Mono.

Listing keys from a bucket in a Riak is a bad idea. This has been documented by Basho and discussed a number of times. Despite this, developers do at some point during development require the ability to list the keys that are in a bucket.

CI should have some documentation which tell the developer that the code that they're using is not code that's intended for production. We can do this via:

• XML documentation on the methods.
• .NET Attributes
• Getting a compile-time warning via abuse of the ObsoleteAttribute
• Clear documentation on ci.org

Inputs can be in one of 4 formats:

1. bucket name
2. array of bucket/keys
3. bucket + key filters
4. {mod: ..., fun: ..., arg: ... }

Refactor the input mechanism to support one of those four mechanisms.

Implement a WriteJson method for each so that this code will be much purdier.

Hello guys

Just been testing out CorrugatedIron, and got some message framing errors - so I had a look into it and had a small refactoring in RiakPbcSocket so that you can ensure you don't get partial network messages:

    private byte[] receiveAll(byte[] resultBuffer)
    {
        int totalBytesReceived = 0;
        int lengthToReceive = resultBuffer.Length;
        while (lengthToReceive > 0)
        {
            int bytesReceived = PbcSocket.Receive(resultBuffer, totalBytesReceived, lengthToReceive, 0);
            if (bytesReceived == 0)
            {
                throw new RiakException("Unable to read data from the source stream - Timed Out.");
            }
            totalBytesReceived += bytesReceived;
            lengthToReceive -= bytesReceived;
        }
        return resultBuffer;
    }

Then, the two receive calls become:

var header = receiveAll(new byte[5]);

and

var resultBuffer = receiveAll(new byte[size - 1]);

Hope that helps.

Matt

Implement Luwak API.

• new MR input phases
• Compile() and MR caching
• delegates for SerDe
• delegates for sibling resolution
• options (RiakDeleteOptions, etc)
• Secondary Indexes - @OJ
• Riak Search - @OJ

This should be implemented.

Riak 1.0 provides many new and magical bucket properties (these can be found in the WebMachine source). CorrugatedIron should expose these.

The FluentInterface takes care of this. We should reduce the public footprint of the library and move contents of CorrugatedIron.Models.MapReduce.KeyFilters to be internal.

Placeholder for discussion about how to best create cross-platform builds for CorrugatedIron.

The current Riak Search API requires that the developer understand more about MR jobs than is really necessary. The API should be improved to make it easier for developers to query Search indexes.

The performance profile will be used to generate a list of known performance pains and to create work items for a 1.1 release.

MR can support search via

    {
     "inputs": {
                "bucket":"mybucket",
                "query":"foo OR bar"
               },
     "query":...
    }```

or

    {
      "inputs": {
                 "bucket":"mybucket",
                 "query":"foo OR bar",
                 "filter":"field2:baz"
                },
      "query":...
    }

In the effort of lowering latency and whatnot, we should change DeleteBucket to use a streaming ListKeys operation and batch up deletes to re-use the same connection.

Remove UpdateLastModified calls from RiakObject. This should be handled by Riak. There is always a chance of clock skew and conditional requests over HTTP may not work.

We need both an agreed upon conflict resolution mechanism and API.

The ServiceStack json serializer is significantly faster than Newtonsoft's: http://www.servicestack.net/benchmarks/

We're looking to use Riak for logging and we need to get past some perf bottlenecks to do this. Faster serialization will be crucial.

Implement Riak Search API.

Using xbuild on OS X generates a bucket of errors related to resolving DLL paths. Similar problems arise when you attempt to use MonoDevelop, too. Not sure if this is related to NuGet based shenanigans, however it makes an automated build outside of Windows a touch more painful.

Gist of output available at https://gist.github.com/4105672

Version info:

XBuild Engine Version 2.10.9.0
Mono, Version 2.10.9.0
Copyright (C) Marek Sieradzki 2005-2008, Novell 2008-2011.

Valid content types will have a slash and be of a specific format. RiakObject should check for valid content types.

The greater than key filter has an incorrect class name of GreatherThan.

Back in the day it was important to provide the facility to handle client IDs when talking to Riak. These days they're ignored by the server.

As of v1.0 of CI, supporting Riak 1.2, we should remove the API calls that deal with Client IDs as they're no longer relevant.