[Question] Is it ok to extend the grpc client class? about java-bigtable-hbase HOT 10 CLOSED

A non-blocking scala client to BigTable sounds very good.

In terms of stability, there are few components to the question.

Protos - We see our proto interfaces are stable. We expect to extend them, but not change them in incompatible ways. The java jar created based on those .proto files will change over time as gRPC's protobuf generator changes, but we're not expecting it to change in incompatible ways.

gRPC - The gRPC project isn't done iterating yet. It's close to having a stable release, probably next week, but it's not there yet. We're expecting that release in the very near future. That said, it's probable that even after the first stable release, we'll be relying on future gRPC SNAPSHOT/development builds for bug fixes and performance purposes.

bigtable-grpc-interface - We haven't thought much about stability around our Java classes. There are some classes that we can mark as stable, and others that are likely to change. I'll mull over how to best mark the differences.

bigtable_hbase - there are a whole bunch of good stuff for reading, specifically filtering on scans, and writing that might be of use. If you stick to the hbase APIs, you're guaranteed the stability that HBase provides.

Perhaps you can describe a bit more about what you're trying to do. That might help us give you more helpful info.

from java-bigtable-hbase.

Top level: I think I just want to make a Scala wrapper around the grpc services.

I haven't thought a great deal about this, but it seems like it would be very easy to replace every method on the bigtable-grpc-interface with one that uses a Twitter Future. This can be done using promises in much the same way. The other thing I'd probably do is make a stream observer that produced a Spool[Row] instead of a List<Row>. A spool is basically a lazy generated list of futures.

I initially thought about extending the class and then just making some implicit to convert between guava futures and exposing both sets of methods, but that seems messy and somewhat silly.

I will whip together a small POC and link back to it here in the next few days.

from java-bigtable-hbase.

Just to clarify one thing: the reason I need a non-blocking client is because I need to issue ~ a thousand reads in under half a second. At the moment I do this with a cassandra cluster + a non-blocking driver here, I don't think the hbase api will work for me in that instance. I'd really like to move to cloud bigtable rather than maintain a cassandra cluster.

from java-bigtable-hbase.

Interesting. Do you want to generate the .java files from the .proto files and muck around with them? I can either generate them and send them over, or walk you through the process of generating them. Changing those generated files might get you what you need to generate a prototype. If it works for you, then long term it might make sense to get those types of changes built into the gRPC .java or .scala file generator.

I would suggest looking at BigtableGrpcClient and ResumingStreamingResultScanner. They might help with understanding some of the gotchas in building resilient and performant readers.

I was able to get a few thousand writes per second from a single process without too much trouble via BufferedMutator.

Let us know how we can help.

from java-bigtable-hbase.

Yeah, mucking about with the gen'd files seems like a good first step. I have cloned the repo and am ready to follow instructions!

from java-bigtable-hbase.

Caveat emptor - this requires you to install protoc 3.0.0 as per https://github.com/grpc/grpc-java. You need gradle as well. (It could be bit of a pain to set up.) From there, you should run "gradle :generateProto" in the bigtable-proto project. Our pom.xml file does the gradle call, but has less debuggability.

That's probably the best way for you to have an end-to-end environment for your efforts. I'll also generate the .java files and post 'em in a branch or something so that you can have a workaround to any protoc / gradle set up issues.

from java-bigtable-hbase.

Wonderful. I'll get all the deps and get cracking.

from java-bigtable-hbase.

The generated protos are at: https://github.com/sduskis/cloud-bigtable-client/tree/generate_protos under bigtable-protos/src/generated-sources

from java-bigtable-hbase.

Thanks, I can no generate the sources with no problems. I'll see what I can get hacked together over the course of this week.

from java-bigtable-hbase.

Closing this.

from java-bigtable-hbase.