I don't know whether this has been fixed. I didn't see a bug in the closed bugs list about this, so just to make sure it doesn't happen again, here is a ticket. Files of the form project/target/config-classes/$<hex number>$.class (e.g. project/target/config-classes/$a25026b6814a5648333d$.class) can be found in 0.5.9 jars from maven. These confuse several tools and make ScalaPB unusable on Android.

Is it possible to support Scala 2.12?

Thanks for creating and open sourcing ScalaPB, love the idea of using the Google proto parser.

I've been using ScalaBuff for a while and contributed some support for hinted deserialization, basically just a map from a class identifier to function to deserialize a byte stream for the matching proto message. Here's an example generated with ScalaBuff:

trait HintedDeserializer {
  def deserializePayload(payload: Array[Byte], payloadType: String): com.google.protobuf.GeneratedMessageLite
}

...

object VoyagerInternalProtos extends HintedDeserializer {
        def registerAllExtensions(registry: com.google.protobuf.ExtensionRegistryLite) {
        }

        private val fromBinaryHintMap = collection.immutable.HashMap[String, Array[Byte] ⇒ com.google.protobuf.GeneratedMessageLite](
                 "VPResult" -> (bytes ⇒ VPResult.parseFrom(bytes)),
                 "VPPlayerIdentifiers" -> (bytes ⇒ VPPlayerIdentifiers.parseFrom(bytes))
        )
        def deserializePayload(payload: Array[Byte], payloadType: String): com.google.protobuf.GeneratedMessageLite = {
                fromBinaryHintMap.get(payloadType) match {
                        case Some(f) ⇒ f(payload)
                        case None    ⇒ throw new IllegalArgumentException(s"unimplemented deserialization of message payload of type [${payloadType}]")
                }
        }
}

SandroGrzicic/ScalaBuff#82

I havent looked behind the scenes for ScalaPB yet, do you think it's reasonable to add something like this to ScalaPB? Could you point me in the right direction where to look & get started?

I imagine for ScalaPB an implementation of a trait along these lines:

trait ScalaPBHintedDeserializer {
  def deserializePayload(payload: Array[Byte], payloadType: String): com.trueaccord.scalapb.GeneratedMessage
}

Any advice appreciated.

Thanks
Peter

Sometimes zero is a good default and Option is just an overhead and one more level of indirection.
Especially, with proto3 making everything optional this is useful in my opinion.

Quickly hacked implementation: eiennohito@a3d20d4

Currently packed enums are not supported. Support should also work for customized enum types.

It seems to be possible according to the docs (https://developers.google.com/protocol-buffers/docs/proto?hl=en#oneof), however codegen generated invalid messages.

for simple message like

syntax = "proto2";

message MapTest {
  map<string, int32> map1 = 1;
  map<string, float> map2 = 2;
}

it generates map entries like case class (key: Option[String], value: Option[Int]), but the type of map is Map[String, Int]. I believe that options are not needed here.

Hi,

Is ScalaPB compatible with ScalaJS?

Thanks

Hi, I wondered if it is possible to generate classes that extend existing (marker) traits? An example:

// General marker for all domain events
trait DomainEvent {
  def persistenceId: String
  def correlationId: UUID
  ...
}

// Specific customer events in a specific package
// Optimally it should be sealed, but that does probably 
// not work because of code gen.

trait CustomerEvent extends DomainEvent { 
  def customerId: ...
  ...
}

// Result of code generation from corresponding .proto files
case class CustomerPolicyChanged(...) extends CustomerEvent
case class CustomerStatusUpgraded(...) extends CustomerEvent
...

When a custom type is specified at the message level, it should be applied for all references to that type.

Hey,

This may not be a bug, so sorry if this is not the right place but I couldn't find a mailing list.

I am investigating ScalaPB as a replacement for sbtprotobuf and pure-Java classes.

When I run the generation command I get this:

> protobuf:protobufGenerate
[info] Compiling 2 protobuf files to /<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf,/<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf
[info] Compiling schema /<snip>/server/src/main/protobuf/abc.proto
[info] Compiling schema /<snip>/server/src/main/protobuf/soundcloud.proto
[error] protoc-gen-scala: program not found or is not executable
[error] --scala_out: protoc-gen-scala: Plugin failed with status code 1.
[trace] Stack trace suppressed: run 'last abc/protobuf:protobufGenerate' for the full output.
[error] (abc/protobuf:protobufGenerate) protoc returned exit code: 1
[error] Total time: 0 s, completed Dec 22, 2014 6:25:37 PM

Full output:

> last abc/protobuf:protobufGenerate
[info] Compiling 2 protobuf files to /<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf,/<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf
[debug] protoc options:
[debug]     --java_out=/<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf
[debug]     --scala_out=/<snip>/server/target/scala-2.10/src_managed/main/compiled_protobuf
[info] Compiling schema /<snip>/server/src/main/protobuf/abc.proto
[info] Compiling schema /<snip>/server/src/main/protobuf/soundcloud.proto
[error] protoc-gen-scala: program not found or is not executable
[error] --scala_out: protoc-gen-scala: Plugin failed with status code 1.
java.lang.RuntimeException: protoc returned exit code: 1
    at scala.sys.package$.error(package.scala:27)
    at sbtprotobuf.ProtobufPlugin$.sbtprotobuf$ProtobufPlugin$$compile(ProtobufPlugin.scala:86)
    at sbtprotobuf.ProtobufPlugin$$anonfun$sourceGeneratorTask$1$$anonfun$6.apply(ProtobufPlugin.scala:110)
    at sbtprotobuf.ProtobufPlugin$$anonfun$sourceGeneratorTask$1$$anonfun$6.apply(ProtobufPlugin.scala:109)
    at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:188)
    at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:188)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:202)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:198)
    at sbt.Difference.apply(Tracked.scala:177)
    at sbt.Difference.apply(Tracked.scala:158)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:198)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:197)
    at sbt.Difference.apply(Tracked.scala:177)
    at sbt.Difference.apply(Tracked.scala:152)
    at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:197)
    at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:195)
    at sbtprotobuf.ProtobufPlugin$$anonfun$sourceGeneratorTask$1.apply(ProtobufPlugin.scala:112)
    at sbtprotobuf.ProtobufPlugin$$anonfun$sourceGeneratorTask$1.apply(ProtobufPlugin.scala:107)
    at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:35)
    at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:34)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
    at sbt.std.Transform$$anon$4.work(System.scala:64)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.Execute.work(Execute.scala:244)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] (abc/protobuf:protobufGenerate) protoc returned exit code: 1

From all this I assume I need to have a plugin called protoc-gen-scala (I thought ScalaPB had its own plugin?). Google takes me to this project: https://github.com/ktoso/protoc-gen-scala, which seems quite old. Either way the build instructions there don't work for me either, so my question becomes: how do I setup protoc-gen-scala to work with ScalaPB?

Here is my plugins.sbt

addSbtPlugin("com.typesafe.sbt" % "sbt-start-script" % "0.10.0")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")

addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")

addSbtPlugin("org.ensime" % "ensime-sbt-cmd" % "0.1.2")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.7.1")

addSbtPlugin("com.trueaccord.scalapb" % "sbt-scalapb" % "0.4.0")

And here are the relevant bits in Build.scala:

/...
import com.trueaccord.scalapb.{ScalaPbPlugin => PB}
/...
lazy val buildSettings = super.settings ++ Defaults.coreDefaultSettings ++ graphSettings ++ microserviceConfig ++ PB.protobufSettings

I've tried versions released before November and had the same issues.

Thanks!

Hi there!

ScalaPB looks awesome, but does it work on wIndows? If not - what should be modified for it to work on windows?

> protobuf:protobufScalaGenerate
[info] Compiling 4 protobuf files to D:\work\scala\tbsgame\target\scala-2.11\src_managed\main\compiled_protobuf
[info] Compiling schema D:\work\scala\tbsgame\protos\base.proto
[info] Compiling schema D:\work\scala\tbsgame\protos\game.proto
[info] Compiling schema D:\work\scala\tbsgame\protos\management.proto
[info] Compiling schema D:\work\scala\tbsgame\protos\messages.proto
[trace] Stack trace suppressed: run last root/protobuf:protobufScalaGenerate for the full output.
[error] (root/protobuf:protobufScalaGenerate) error occured while compiling protobuf files: Cannot run program "mkfifo": CreateProcess error=2, The system cannot find the file specified
[error] Total time: 0 s, completed 2015-03-25 15.39.13

Also - is there a way to use it in a standalone fashion, not as a part of SBT?

ScalaPB makes beautiful case classes, but they include an extra val called serializedSize. This field makes it impossible to use a library like spray-json to automatically generate JSON formatters for these case classes. Marking the serializedSize field as @transient will resolve this issue.

https://github.com/trueaccord/ScalaPB/blob/425dd2d52a2261c5e63e392243bae8e0c97fe3e8/compiler-plugin/src/main/scala/com/trueaccord/scalapb/compiler/ProtobufGenerator.scala#L400

For projects that don't care about the Java bindings, it should be possible to get ScalaPB to use the declared package instead of generating a different one.

Currently there is no built-in serializer to convert messages ↔ Json. This could come in handy to send a readable Json to the client.

Would it make sense to have com.trueaccord.scalapb.GeneratedMessage and com.trueaccord.scalapb.GeneratedEnum extend java.io.Serializable. Reason for asking is that akka serialization picks the wrong serializer due to this:

In case of ambiguity, i.e. the message implements several of the configured classes, the most specific configured class will be used, i.e. the one of which all other candidates are superclasses. If this condition cannot be met, because e.g. java.io.Serializable and MyOwnSerializable both apply and neither is a subtype of the other, a warning will be issued

I believe com.google.protobuf.GeneratedMessage implements java.io.Serializable with serialVersionUID = 1L.

Because otherwise debugging of non-english applications become a nightmare and world is not just English.

An option in the build would be a solution as well.

Use case: Exhaustive pattern matching of related domain events.

A file level option like:
option (scalapb.message).sealedTrait = "DomainEvent";

This would cause all messages in the .proto file to be put in the same .scala file along with a
sealed trait DomainEvent {}

When the field is named get:

[error] overloaded method get needs result type
[error] def get = field(_.getGet)((c_, f_) => c_.copy(get = Some(f_)))

When the field is named tag:

[error] reassignment to val
[error] __tag = Some(__input.readString())

When IntelliJ imports an SBT project using the ScalaPB plugin it adds both directories:
"src_managed / main" and "src_managed / main / compiled_protobuf" to it's internal build system.
So when you build the project with IntelliJ the generated classes get included twice, causing conflicts.

is it possible to build directly into the "src_managed / main" directory ?
sbt-protobuf has some instructions about this, but it's not obvious how they map to ScalaPB.

Can you make an option for it to use Apache commons-codec or something for base64 decoding?

I did run (on windows) into the issue of too long file names which caused the compiler to fail writing the InternalFields_xxx.scala document.

While it is primarily Windows to blame for the last century 256 path length limitation, scalaPB seems to generate filenames that are unnecessarily long: xxx contains the full path (from the protobuf root), but since the files are stored in the scala-package directory, using the .proto-filename alone should be sufficient. In my case this would have prevented the error from happening as my protobuf files reside in a quite deep file-hierarchy.

I could workaround the problem by changing my package-structure (shorter names), so no show stopper.

More important than this fix would be a good error message that indicates what went wrong. It took me quite some time to figure out what's going on.

Nevertheless: Thanks a lot for providing and supporting scalaPB!

As far as I can tell, Protocol Buffer Extensions aren't supported by ScalaPB. Is it possible to add support?

In proto3, when the proto file name contains a message with identical name, the Java outer class name becomes "${file}OuterClass", this current breaks the java conversion feature.

Hello. My env scala 2.11 protobuf 3 alpha

syntax = "proto3";

package grpc.playground.helloworld;

option java_multiple_files = true;
option java_package = "grpc.playground.helloworld";
option java_outer_classname = "HelloWorldProto";

// The greeting service definition.
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloResponse) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloResponse {
  string message = 1;
}

When i compiled it, java compiler show me this

HelloRequest.scala:50: type HelloRequest is not a member of object grpc.playground.helloworld.HelloWorldProto

For every class. At first I thought it was error with package, but when i comment class_name in protofie, it appeared again

hello-world/target/scala-2.11/src_managed/main/compiled_protobuf/grpc/playground/helloworld/hello_world/HelloRequest.scala:50: type HelloRequest is not a member of object grpc.playground.helloworld.HelloWorld
[error] object HelloRequest extends com.trueaccord.scalapb.GeneratedMessageCompanion[HelloRequest] with com.trueaccord.scalapb.JavaProtoSupport[HelloRequest, grpc.playground.helloworld.HelloWorld.HelloRequest]  {

When i turn off java conv it all work fine. It's bug?

google/protobuf/descriptor.proto is both in scalapb-runtime and in protobuf-java jars. This is a problem when using ScalaPB on Android.

Hi,

First, thanks for the great work !

I'm wondering why this :
message my_message { optional uint32 foo = 1 [ default = 0 ]; }

results in foo being an Option[Int], not an Int. It will always have a value right ?

Cheers

Would it be possible for messages with one required field of value type to extend AnyVal? I am presuming not, but it's not completely clear from a cursory scan of the code.

https://travis-ci.org/trueaccord/ScalaPB/jobs/91336480#L1442

[error] /home/travis/build/trueaccord/ScalaPB/e2e/target/scala-2.10/src_managed/main/compiled_protobuf/com/trueaccord/pb/flat/FlatProto.scala:20: object Scalapb is not a member of package com.trueaccord.scalapb
[error]   com.trueaccord.scalapb.Scalapb.getDescriptor(),
[error]                          ^
[error] /home/travis/build/trueaccord/ScalaPB/e2e/target/scala-2.10/src_managed/main/compiled_protobuf/com/trueaccord/proto/e2e/custom_types/CustomTypesProto.scala:42: object Scalapb is not a member of package com.trueaccord.scalapb
[error]   com.trueaccord.scalapb.Scalapb.getDescriptor()
[error]                          ^
[error] /home/travis/build/trueaccord/ScalaPB/e2e/target/scala-2.10/src_managed/main/compiled_protobuf/com/trueaccord/proto/e2e/maps/MapsProto.scala:30: object Scalapb is not a member of package com.trueaccord.scalapb
[error]   com.trueaccord.scalapb.Scalapb.getDescriptor()
[error]                          ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed

In some cases zeros does not make sence. For example, I have a model Event, stored in relational database:

message Event {
  bytes data = 1;
  int64 deleted_at = 5 [(scalapb.field).type = "org.joda.time.DateTime"];
}

In the past Event did not have a deleted_at, but it had a boolean is_deleted. When deleted_at appeared, it had been set to the epoch start for every deleted event and to NULL for the rest. So, if deleted_at is not an Option, there is no way to determine if event was deleted.

I'd like to have an ability to exclude some protofiles from code generation. I tried to add:

excludeFilter in unmanagedSources := "filename.proto" 

to build.sbt, but it is ignored.

If I want to write a seq of protos to a file (rather than a single proto), I would expect to need to use writeDelimitedTo. Has this simply not yet been implemented?

See: protoc at https://github.com/sbt/sbt-protobuf#settings

Looks like "protoc" is hardcoded here https://github.com/trueaccord/ScalaPB/blob/57695392abbca236e9121cc6fad624200e29b6f1/compiler-plugin/src/main/scala/com/trueaccord/scalapb/compiler/Process.scala#L142

org/a/a.proto

syntax = "proto2";
package org.a;

message Msg {
extensions 16 to max;
}

org/b/b.proto

syntax = "proto2";
package org.b;
import "org/a/a.proto";

extend Msg {
optional Test test = 16;
}

message Test {}

These configuration throws error while compiling:

protoc-jar: executing: [C:\Users\D\AppData\Local\Temp\protoc3931572059064664031.exe, --plugin=protoc-gen-scala=C:\Users\D\AppData\Local\Temp\scalapbgen2982993506159287014.b
at, -ID:\my_app\app, -ID:\my_app\target\protobuf_external, --scala_out=:D:\my_app\target\scala-2.11\src_managed\main
\compiled_protobuf, D:\my_app\app\org\a\a.proto,
D:\my_app\app\org\b\b.proto]
org/b/b.proto:6:8: "Msg" is not defined.
[libprotobuf WARNING google/protobuf/descriptor.cc:6117] Warning: Unused import: "org/b/b.proto" imports "org/a/a.proto" which is not used.
java.lang.RuntimeException: protoc returned exit code: 1

After generating pb files, I cannot rebuild the project with this error

Error:(10, 18) LoginFailed is already defined as case class LoginFailed
final case class LoginFailed(

Error:(56, 8) LoginFailed is already defined as object LoginFailed
object LoginFailed extends com.trueaccord.scalapb.GeneratedMessageCompanion[LoginFailed] {

I cannot compile a simple oneof message:

message Test {
  oneof haha_type {
    string a = 1;
    string b = 2;
  }
}

with the errors

[error] Expected "required", "optional", or "repeated".
[error] Missing field number.

We're calling onComplete after onError when converting a call's result from a Try to a StreamObserver. This violates the interface contract of StreamObserver, which says onError should be the last method called: https://github.com/grpc/grpc-java/blob/master/stub/src/main/java/io/grpc/stub/StreamObserver.java#L63

This causes the stream to be double-closed, resulting in a IllegalStateException when an error occurs inside of a gRPC service method.

http://www.grpc.io/

I am unable to add this plugin addSbtPlugin("com.trueaccord.scalapb" % "sbt-scalapb" % "0.4.20") to my build due to issue in sbt-protobuf: see sbt/sbt-protobuf#37 for details.

error: reference to compile is ambiguous;
it is imported twice in the same scope by
import _root_.sbtprotobuf.ProtobufPlugin._
and import Keys._
  settings(mainClass in compile := Some("whatever")

If we are trying to parse a message with an unknown enum value, mergeFrom raises a MatchError.

Currently containingOneOfName returns something like:
"Some(com.google.protobuf.Descriptors$OneofDescriptor@63eb32ed)"

The reason is because the oneof descriptor is being passed in instead of the name of the oneof descriptor.

https://github.com/trueaccord/ScalaPB/blob/a6cd42a9904ce1f8f9fa010df6da43676d0eeb66/compiler-plugin/src/main/scala/com/trueaccord/scalapb/compiler/ProtobufGenerator.scala

Line 790

Should be:
val oneof = field.containingOneOf.map(s => s"""Some("${s.getName()}")""").getOrElse("None")

scalapb/scalapb.proto contains a syntax="proto2"; directive in it, this directive isn't understood by protoc (2.x.x) This means using scalapb's very nice advance features isn't possible. Our team isn't ready to move to 3.x given it's alpha status but we'd like to start using scalapb's advance field type functionality.

The Descriptors appears to be missing the functionality to access the field/enum/message options.

Hey Guys,

Thanks for the awesome lib. On this page http://trueaccord.github.io/ScalaPB/customizations.html you misspelled libraryDependencies (you're missing the d)

libraryDepenencies += "com.trueaccord.scalapb" %% "scalapb-runtime" % "0.4.9" % PB.protobufConfig

Since I have all my messages as .proto I would like to take advantage of that and generate all boiler plate code for my application.

Examples of generated code other than plain case classes:

• Custom JSON ser / dser
• Slick table models
• Web forms with input validation

What I'm looking for here is an easy way of plugging my own custom code generator into ScalaPB's pipeline.

Any thoughts on that? How hard is it to do this today if I fork the project?

Proto3 has standard classes for wrapping primitives: https://github.com/google/protobuf/blob/master/src/google/protobuf/wrappers.proto.

Can we provide a way for the generator to map these types to optional primitives (e.g., Int32Value maps to Option[Int])?

println(B.parseFrom(A("hello", 100).toByteArray)) // B(0, "")
println(A.parseFrom(B(100, "hello").toByteArray)) // A("", 0)

Is this intended? Then why do we have a "validate" method? Is there a sbt setting to change these defaults and make it throw exception?