raydac / java-binary-block-parser Goto Github PK

Hello,
I would like to know if there is a way to use several customer type processors (example: one for float, one for String) in the JBBPParser.
It seems that the "prepare" method accepts only 1 custom parser.
Thank you in advance for your help.
Best regards,
Bruno

Hello,
JBBPNumericField does not to implement float Type ( getAsfloat() ).
This seems to be mandatory before writing custom type processor for float values ?

is this analysis correct ?

thks

This would maybe entail the ability to read a kaitai file and create the appropriate java object with annotations and JBBPOut.BeginBin() block for reading a writing files.

Thanks.

When I use a child class for parsing will throw java.lang.IllegalStateException: There is not any opened struct. Below are my test code.
@bin
@getter
@Setter
public class Parent {
@bin(order = 1, type = BinType.USHORT, comment = "Size of package header")
int headerSize;
}

@bin
@getter
@Setter
public class Child extends Parent {
@bin(order = 2, type = BinType.BYTE_ARRAY, arraySizeExpr = "1024")
String body;
}

public class Test {

@Test
public void testParse() {
    byte[] data = new byte[] {0x12, 0x34, 0x56, 0x7F};
    final JBBPParser parser = JBBPParser.prepare(JBBPDslBuilder.Begin().AnnotatedClass(Child.class).End());
    JBBPFieldStruct parsed = parser.parse(data);

    final MenuPackageTwo parsedPackage = parsed.findFieldForNameAndType("Child", JBBPFieldStruct.class).mapTo(new Child());

}
}

Hi Igor,

Have you thought about how best to handle Strings within the binary data? I realize this could be done with a JBBPCustomFieldTypeProcessor implementation, but it seems like a common enough use case to warrant inclusion in the DSL. For example, in the binary data I'm currently working with, there are things like serial numbers or vehicle identification numbers encoded as ASCII strings within the binary message. Usually there's a length indicator (ubyte), followed by the ASCII encoded data. Of course, it would be good to include an option on the JBBPParser to specify the character set you want to use when parsing the string type (defaulting to UTF-8, which should work most of the time). The "extra data" could be used to indicate the length of the string. For example:

serial_number {
  ubyte length;
  str:(length);
}

As always, I'm willing to help in whatever way I can.

Hello
Thank you very much for your library. I just discovered it and it might help for my project. Up to now I was able to fully read the bytes from my device. Now I want to try to send data back to the device. So far everything works as designed.

Unfortunately the device has some packages containing a version when reading but we must not send them when writing to the device.
Example Bin annotated class:

public class Foo {
   @Bin(order = 0, type = BinType.BYTE)
    protected int startByte = 0x02;
    @Bin(order = 1, type = BinType.BYTE)
    protected int length;
    @Bin(order = 2, type = BinType.BYTE)
    protected int key;
    @Bin(order = 3, type = BinType.BYTE) // must be removed when sending
    protected int packageVersion;
    @Bin(order = 4, type = BinType.BYTE)
   // more fields here
}

When sending to the device I must remove the field packageVersion. Is there a way to have something like the following?

BeginBin().Bin(new Foo()).RemoveFieldByName("packageVersion").End();
BeginBin().Bin(new Foo(), filterDefinition).End();

Or at class level using Bin annotation:

@Bin(process = "read")
@Bin(process = "write")
@Bin(process = "always") // default

Is there already another way achieving this, or do I need to write a custom writer?
Best regards,
Tom

I have a parent class:

import com.igormaznitsa.jbbp.mapper.Bin;

public class BreakJBBPDslBuilderParent {
    @Bin(outOrder = 1)
    public BreakJBBPDslBuilderChild[] breakJBBPDslBuilderChildArray;
}

And a child class:

import com.igormaznitsa.jbbp.mapper.Bin;
import com.igormaznitsa.jbbp.mapper.BinType;

public class BreakJBBPDslBuilderChild {
    @Bin(outOrder = 1, comment = "Reserved", type = BinType.BYTE)
    public byte reserved;
}

If I try to use the JBBPDslBuilder on the parent class like this:

JBBPDslBuilder.Begin().AnnotatedClass(BreakJBBPDslBuilderParent.class).End();

I get an IllegalArgumentException with the message "Expression is empty" that is thrown by this line:

Is this the correct way I'd use the JBBPDslBuilder to work with a structure that has arrays in it?

Plugin version: 2.0.4
Gradle: 7.4.2

I am trying to apply the JBBP plugin to a simple empty command line project and get this error.

Stacktrace:

Caused by: java.lang.IllegalArgumentException: Cannot add extension with name 'jbbp', as there is an extension already registered with that name.
	at org.gradle.internal.extensibility.ExtensionsStorage.add(ExtensionsStorage.java:40)
	at org.gradle.internal.extensibility.DefaultConvention.add(DefaultConvention.java:120)
	at org.gradle.internal.extensibility.DefaultConvention.create(DefaultConvention.java:138)
	at org.gradle.internal.extensibility.DefaultConvention.create(DefaultConvention.java:132)
	at com.igormaznitsa.jbbp.plugin.gradle.JBBPPlugin.apply(JBBPPlugin.java:14)
	at com.igormaznitsa.jbbp.plugin.gradle.JBBPPlugin.apply(JBBPPlugin.java:7)

I quickly checked out the source code here and I indeed see double creation of extension.

Hello,

from what I read in your documentation, I do not see the way to map a primary type on an abritrary bit/byte lenght. Like how could I do for:

• 24 bits signed integer (so stored on 3 bytes)
• 23 bits signed integer (so stored on 2 bytes + 7 bits)

Every primary types should support the notation [:1..x] not only bit type, no?

Thank you for you help,
Regards,
Cedric,

It is not an issue, I just want to discuss with you.

I found this project on stackoverflow, it was impressive to me because I also developed a little similar project(FastProto).

Unlike you(python struct), I work in IoT development, and exchanges between devices and servers often use binary formats, so I developed FastProto hoping to be able to parse binary and pack binary quickly.

Although it has been in development for three years, the project is not very popular, and I often don't have much confidence in my technology. If possible, could you suggest some improvements to the project.

https://github.com/indunet/fastproto

the prepare method can take a custom type processor. Is there a way to have it take multiple custom type processors? say we have two different custom types.

thanks!

Works in gradle 6.7.1
Gives error in gradle 7.4.2

> Task :myTask:jbbpGenerate FAILED

FAILURE: Build failed with an exception.

What went wrong:
* A problem was found with the configuration of task ':myTask:jbbpGenerate' (type 'JBBPGenerateTask').
- In plugin 'com.igormaznitsa.gradle.jbbp' type 'com.igormaznitsa.jbbp.plugin.gradle.JBBPGenerateTask' field 'addSource' without corresponding getter has been annotated with @Input, @Optional.

Reason: Annotations on fields are only used if there's a corresponding getter for the field.

Possible solutions:
  1. Add a getter for field 'addSource'.
  2. Remove the annotations on 'addSource'.

Please refer to https://docs.gradle.org/7.4.2/userguide/validation_problems.html#ignored_annotations_on_field for more details about this problem.

Hi,

I would like to parse a binary stream of complex, nested structures, like it is done e.g. in PNGParsingTest, where the class Png has a non-primitive member of type Chunk.

In the test file, the description of the binary format (the arguement to prepare) is given as a complete, nested structure, together with the two class declarations in the same file.

I would prefer to keep the declaration of the two classes separate together with their binary description, and only make a reference to the Chunk type in the DSL for the Png class and have JBBP resolve the reference. Is that possible with the current state of JBBP?

Also, is there any way to automatically inherit from a binary parser description to reuse it for an inherited mapped class?

hello:

byte[] data,is generated by JBBPOut, which use JBBPByteOrder.LITTLE_ENDIAN;
how to parse ? I can not find any test case in JBBPMapperTest.java

thanks

Thanks for your library. We are upgrading from 1.3 version of your library and we have custom field parsers for explicit ASCII/ISO-8859-1 and UTF-8 strings.

From your implementation, I see "stringj" works just for UTF-8. Is my understanding correct and we should keep our own parsers if we want explicit support for other codecs?

class Parsed {
        @Bin (type = BinType.BYTE_ARRAY) String str;
      }
      Parsed p = JBBPParser.prepare("byte [_] str;").parse(new byte[]{49,48,47}).mapTo(Parsed.class);
      System.out.println(new JBBPTextWriter().Bin(p).toString());

throws

Exception in thread "main" java.lang.NullPointerException
    at com.igormaznitsa.jbbp.utils.JBBPTextWriter$MappedObjectLogger.onArrayStart(JBBPTextWriter.java:133)
    at com.igormaznitsa.jbbp.io.AbstractMappedClassFieldObserver.processObjectField(AbstractMappedClassFieldObserver.java:326)
    at com.igormaznitsa.jbbp.io.AbstractMappedClassFieldObserver.processObject(AbstractMappedClassFieldObserver.java:161)
    at com.igormaznitsa.jbbp.utils.JBBPTextWriter$MappedObjectLogger.processObject(JBBPTextWriter.java:279)
    at com.igormaznitsa.jbbp.utils.JBBPTextWriter.Bin(JBBPTextWriter.java:1528)
    at com.igormaznitsa.testdi.Main.main(Main.java:15)

Hi,

Does this library load the entire read binary file into memory ?

Thanks

I am trying to parse binary file which begins with string (3 chars) with null termination.

The problem is, that this library requires to have prefix '3' before string, but in my binary file here is not this type of prefix.

My question how to parse string when here is no '3' prefix in front of string?

Failing error:

2021-11-09 12:50:22.141 11264-11264/com.example.binaryparser E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.binaryparser, PID: 11264
    java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.binaryparser/com.example.binaryparser.MainActivity}: java.lang.NullPointerException: Attempt to invoke virtual method 'com.igormaznitsa.jbbp.model.JBBPAbstractField com.igormaznitsa.jbbp.model.JBBPFieldStruct.findFieldForType(java.lang.Class)' on a null object reference

Binary file structure is:

    let prefix: String
    let checksum: UInt32
    let version: ECGFileVersion
    let reserved: UInt8
    let channelMask: UInt16
    let sampleCount: UInt32
    let frequency: UInt32
    let reference: Int32
    let gains: [Int32]
    var samples: [Int] = [Int]()

To pars file prefix using this code:

        final JBBPParser parser = JBBPParser.prepare("stringj;");
        JBBPFieldStruct result = null;
        try {
            result = parser.parse(data);
        } catch (IOException e) {
            e.printStackTrace();
        }
        String dd = result.findFieldForType(JBBPFieldString.class).getAsString();

I have binary file like this:

Access is allowed from event dispatch thread only.

EventQueue.isDispatchThread()=false isDispatchThread()=false Toolkit.getEventQueue()=com.intellij.ide.IdeEventQueue@12bc02ff Current thread: Thread[ApplicationImpl pooled thread 6,4,Idea Thread Group] 573120594 SystemEventQueueThread: Thread[AWT-EventQueue-0,6,main] 1272330861

java.lang.Throwable
at com.intellij.util.ExceptionUtil.currentStackTrace(ExceptionUtil.java:65)
at com.intellij.psi.impl.DebugUtil.currentStackTrace(DebugUtil.java:486)
at com.intellij.diagnostic.LogEventException.(LogEventException.java:32)
at com.intellij.openapi.application.impl.ApplicationImpl.b(ApplicationImpl.java:1089)
at com.intellij.openapi.application.impl.ApplicationImpl.assertIsDispatchThread(ApplicationImpl.java:1078)
at com.intellij.openapi.command.impl.CoreCommandProcessor.a(CoreCommandProcessor.java:130)
at com.intellij.openapi.command.impl.CoreCommandProcessor.executeCommand(CoreCommandProcessor.java:109)
at com.intellij.openapi.command.impl.CoreCommandProcessor.executeCommand(CoreCommandProcessor.java:99)
at com.intellij.openapi.command.impl.CoreCommandProcessor.executeCommand(CoreCommandProcessor.java:85)
at com.igormaznitsa.ideamindmap.utils.IdeaUtils.findKnowledgeFolderForModule(IdeaUtils.java:244)
at com.igormaznitsa.ideamindmap.facet.MindMapFacet.(MindMapFacet.java:38)
at com.igormaznitsa.ideamindmap.facet.MindMapFacetType.createFacet(MindMapFacetType.java:41)
at com.igormaznitsa.ideamindmap.facet.MindMapFacetType.createFacet(MindMapFacetType.java:28)
at com.intellij.facet.FacetManagerImpl.createFacet(FacetManagerImpl.java:136)
at com.intellij.facet.FacetManagerImpl.a(FacetManagerImpl.java:275)
at com.intellij.facet.FacetManagerImpl.a(FacetManagerImpl.java:212)
at com.intellij.facet.FacetManagerImpl.loadState(FacetManagerImpl.java:294)
at com.intellij.facet.FacetManagerImpl.loadState(FacetManagerImpl.java:58)
at com.intellij.configurationStore.ComponentStoreImpl.a(ComponentStoreImpl.kt:328)
at com.intellij.configurationStore.ComponentStoreImpl.a(ComponentStoreImpl.kt:290)
at com.intellij.configurationStore.ComponentStoreImpl.initComponent(ComponentStoreImpl.kt:90)
at com.intellij.openapi.components.impl.PlatformComponentManagerImpl.initializeComponent(PlatformComponentManagerImpl.java:54)
at com.intellij.openapi.components.impl.ComponentManagerImpl$ComponentConfigComponentAdapter.getComponentInstance(ComponentManagerImpl.java:495)
at com.intellij.openapi.components.impl.ComponentManagerImpl.createComponents(ComponentManagerImpl.java:119)
at com.intellij.openapi.components.impl.ComponentManagerImpl.init(ComponentManagerImpl.java:103)
at com.intellij.openapi.module.impl.ModuleImpl.init(ModuleImpl.java:73)
at com.intellij.openapi.module.impl.ModuleManagerImpl$ModuleModelImpl.a(ModuleManagerImpl.java:806)
at com.intellij.openapi.module.impl.ModuleManagerImpl$ModuleModelImpl.e(ModuleManagerImpl.java:800)
at com.intellij.openapi.application.ReadAction.compute(ReadAction.java:47)
at com.intellij.openapi.module.impl.ModuleManagerImpl$ModuleModelImpl.c(ModuleManagerImpl.java:798)
at com.intellij.openapi.module.impl.ModuleManagerImpl$ModuleModelImpl.access$400(ModuleManagerImpl.java:638)
at com.intellij.openapi.module.impl.ModuleManagerImpl.a(ModuleManagerImpl.java:301)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:173)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:157)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:543)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:488)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:94)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:144)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:173)
at com.intellij.openapi.module.impl.ModuleManagerImpl.a(ModuleManagerImpl.java:299)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.intellij.util.concurrency.BoundedTaskExecutor$2.run(BoundedTaskExecutor.java:212)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Hi Igor,

It would be helpful to specify the "extra data" portion of a field with an expression. If you agree it would be useful, I'd be happy to take a first cut and submit another pull request.

Here's the reason for the request. BCD is something that belongs at the field level...it's a single entity within the binary message. However, the length of the field varies, and in some message formats there's a length field prior to the BCD bytes. For example:

{<bcd length ubyte>, <bcd field byte 0>, ... <bcd field byte (length-1)>}

I'd like to be able to specify that with the DSL...something like this:

ubyte length;
bcd(length) bcd_field;

Or for an array:

ubyte field_count;
ubyte field_length;
bcd(field_length) [field_count] bcd_fields;

The exact syntax doesn't really matter, as long as it's possible to specify the field size via an expression.

Thoughts?

Hi Igor,

Here's another feature to consider. It would be very cool (IMHO) to have support for parsing sections of a script only if a conditional expression evaluates to true (the condition being determined from data that was already parsed).

For example, the message format I have been working with is documented here. This message format has a lot of optional fields or sets of fields, with a bit or byte indicating whether they are included.

The first byte of the message indicates whether the "options header" is present (msb set), and if so, what other sets of fields (all optional) are present within the header. There's an "options extension" section within the options header, that includes additional optional fields. The message header then has a field to indicate which message type will follow, and the message types also have optional sets of fields indicated by bits. It would be really cool if a single parse script could describe this type of message, with conditional expressions determining which sections of the script get parsed, according to what is there.

Just FYI, here's how I accomplished parsing a message type using the library (plus my extensions for strings, packed decimal, conversion to map):

1. Create the JBBPBitInputStream wrapper around the ByteArrayInputStream outside of the parser so it could be reused by subsequent parsers while keeping the correct position
2. Parse the first byte into a struct.
3. If msb is set, parse the option header as follows:
   1. Determine which data sections are within the options header, as indicated by other bits in the first byte
   2. Concatenate together the parse scripts for the data sections that are present into a single string
   3. Parse the rest of the options header, except the options extension
   4. Repeat the above conditional parsing strategy for the options extension (if present)
4. If msb is not set, reset() the input stream and start with the message header
5. Parse the message header
6. Determine the message type from the message header
7. Parse the message type (including optional fields -- e.g., "extension strings" in message type 3)

All of that requires 5 separate calls to JBBPParser.prepare(...).parse(...) to parse a message, but with conditional parsing it could be done with a single parse.

I was thinking that conditional expressions could be implemented at the struct level within the DSL syntax, similar to how expressions can be used to calculate the extra data for fields. The script to parse the options header might look something like this:

options_header {
  options_byte {
    bit mobile_id;
    bit mobile_id_type;
    bit auth_word;
    bit routing;
    bit forwarding;
    bit response_redir;
    bit options_ext;
    bit present;
  }
  reset:(!present) {
    // reset the options_byte just read
    reset$$:1;
  }
  mobile_id:(present && mobile_id) {
    ubyte mobile_id_length;
    bcd:(mobile_id_length) mobile_id;
  }
  mobile_id_type:(present && mobile_id_type) {
    ubyte mobile_id_type_length; 
    ubyte mobile_id_type;
  }
  authentication_word:(present && auth_word) {
    ubyte authentication_length; 
    int authentication;
  }
  routing:(present && routing) {
    ubyte routing_length; 
    byte[routing_length] routing;
  }
  forwarding:(present && forwarding) {
    ubyte forwarding_length;
    byte[forwarding_length] forwarding;
  }
  response_redirection:(present && response_redir) {
    ubyte response_redirection_length;
    ubyte[4] response_redirection_addr;
    ushort response_redirection_port;
  }
  options_extension:(present && options_ext) {
    ubyte options_extension_length;
    bit esn_enabled;
    bit vin_enabled;
    bit encryption_service_enabled;
    align;
    options_extension_esn:(esn_enabled) {
      ubyte esn_length;
      bcd:(esn_length) esn;
    }
    options_extension_vin:(vin_enabled) {
      ubyte vin_length;
      str:(vin_length) vin;
    }
    options_extension_encryption:(encryption_service_enabled) {
      ubyte encryption_length;
      byte[encryption_length] encryption_service_bytes;
    }
  }
}

I just tested to generate java objects from a binary stream, which worked for me after annotating my fields with @bin(type=BinType.UBYTE) because it doesn't automatically map it to integers. But now I am looking for the opposite direction. Get the binary stream from my class, but I can't seem to find it. Is it implemented?

JBBPOut use the Object's @bin annotation ByteOrder instead of inherit the ByteOrder from JBBPOut constructor when call JBBPOut.Bin(Object)

example:

`

   byte[] bins = JBBPOut.BeginBin(JBBPByteOrder.LITTLE_ENDIAN).Int(123).Bin(new CommandData(123)).End().toByteArray();
   //actually, bins=[123, 0, 0, 0, 0, 0, 0, 123], but i want bins=[123, 0, 0, 0, 123, 0, 0, 0]

 static class CommandData{
    @Bin(order = 1)
    int data;

    public CommandData(int data) {
        this.data = data;
    }
}

`

Yes, a readme contains a link t example of defining some unsigned integer. But why do not include it? As user I have to copy this from different sources on Internet. Unsigned int is needed. You have a lot experience with JBBP - adding unsigned int32 will not take a lot of time.

I have multiple classes that I want to convert into the DSL. Most of them work but I found a few that didn't and failed with NullPointerExceptions.

The simplest example I could come up with was:

import com.igormaznitsa.jbbp.mapper.Bin;
import com.igormaznitsa.jbbp.mapper.BinType;

public class BreakJBBPDslBuilder {
    @Bin(outOrder = 1, comment = "Reserved", type = BinType.BIT_ARRAY, extra = "4")
    public byte[] reserved;
}

If I then do this:

JBBPDslBuilder.Begin().AnnotatedClass(BreakJBBPDslBuilder.class).End();

I'll get a NullPointerException in the JBBPDslBuilder class on line 1936.

In this case field.bin.outBitNumber is equal to BITS_8 so it then tries to execute the this.bin.outBitNumber() method. this.bit is the field that is NULL. I went through the test suite and it looks like the this.bin field is always NULL in the tests. Is it really needed? I haven't fully understood all of the code but maybe this should've been a NULL check on field.bin?

I'm working with esp 32 and have task to unpack c structure on Android.
With python & struct it's very easy
struct.unpack("<cffhhhhfffccc", message)
But I have failed to make it with this library. To make it work with this lib I need to unpack to integer using "<int" and then use Float.intBitsToFloat to this value. It's not so comfortable and readable code when you have a lot of structure fields.
Could you add "<floatj" behaves same as in python struct packet, please?
Expected behavior
Byte array - 00 00 80 3f
parse with "<floatj" give 1.0
actual behavior
parse with "<floatj" give

I've written a JBBPCustomFieldTypeProcessor to parse uint32 fields into Java longs: https://gist.github.com/use-sparingly/ac1539784af7ae7569cd50135016cf2c

It seems to work for single, read all ([_]) and defined length arrays ([1]). However, if I try to reference the value as a field length in a subsequent field I get a nullpointer exception.

Example:

        final JBBPParser sasParser = JBBPParser.prepare(
                ">uint32 keycount;" +
                "key [keycount] {" +
                        "byte[1] contentId; " +
                        "byte[1] keyData; " +
                "}"
        , new Uint32()
        );

        final String inputSHex = "00000001fc05";
        byte inputS[] = Hex.decodeHex(inputSHex.toCharArray());

        final JBBPFieldStruct result = sasParser.parse(inputS);

I get:

Exception in thread "main" java.lang.NullPointerException
	at com.igormaznitsa.jbbp.compiler.varlen.JBBPOnlyFieldEvaluator.eval(JBBPOnlyFieldEvaluator.java:52)
	at com.igormaznitsa.jbbp.JBBPParser.parseStruct(JBBPParser.java:174)
	at com.igormaznitsa.jbbp.JBBPParser.parse(JBBPParser.java:489)
	at com.igormaznitsa.jbbp.JBBPParser.parse(JBBPParser.java:515)
	at Main.main(Main.java:31)

If I replace uint32 with int then it works as expected (until I use a true uint32 value).

Hi, raydac:

When I use this lib on my personal Android App with version '1.4.0', I got an error like below.

Process: com.demo.netsniffer, PID: 1646 java.lang.ExceptionInInitializerError at com.igormaznitsa.jbbp.compiler.JBBPCompiler.compile(JBBPCompiler.java:214) at com.igormaznitsa.jbbp.JBBPParser.<init>(JBBPParser.java:132) at com.igormaznitsa.jbbp.JBBPParser.prepare(JBBPParser.java:160) at com.igormaznitsa.jbbp.JBBPParser.prepare(JBBPParser.java:206) at com.demo.netsniffer.core.SnifferVpnService.<init>(SnifferVpnService.java:38) at java.lang.Class.newInstance(Native Method) at android.app.ActivityThread.handleCreateService(ActivityThread.java:3173) at android.app.ActivityThread.-wrap5(ActivityThread.java) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1567) at android.os.Handler.dispatchMessage(Handler.java:102) at android.os.Looper.loop(Looper.java:154) at android.app.ActivityThread.main(ActivityThread.java:6119) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776) Caused by: java.util.regex.PatternSyntaxException: Syntax error in regexp pattern near index 14 \s*//.*$|\s*(})|\s*([^\s;\[\]}{]+)?\s*(?:\[\s*([^\[\]{};]+)\s*])?\s*([^\d\s;\[\]}{/][^\s;\[\]}{/]*)?\s*([{;]) ^ at java.util.regex.Pattern.compileImpl(Native Method) at java.util.regex.Pattern.compile(Pattern.java:1340) at java.util.regex.Pattern.<init>(Pattern.java:1324) at java.util.regex.Pattern.compile(Pattern.java:971) at com.igormaznitsa.jbbp.compiler.tokenizer.JBBPTokenizer.<clinit>(JBBPTokenizer.java:46) ... 15 more

It seems that the regex pattern in "JBBPTokenizer.java" can not be complied on Android.
field:
private static final Pattern PATTERN = Pattern.compile("\\s*//.*$|\\s*(})|\\s*([^\\s;\\[\\]}{]+)?\\s*(?:\\[\\s*([^\\[\\]{};]+)\\s*])?\\s*([^\\d\\s;\\[\\]}{/][^\\s;\\[\\]}{/]*)?\\s*([{;])", Pattern.MULTILINE);

Could you fix it?

Hi
I tried to use JBBP to resolve DNS query messages, but I don't know what to write. Can you provide an example.

Hi developer:

I use JBBP to parse the signals generated from IOT device , the device generate signal in BIG_ENDIAN and MSB0 bitorder:
but when I parse some string from the signal, I got Error result:

the test code is below:

public static void main(String[] args) {
// TODO Auto-generated method stub
long startTime = System.currentTimeMillis();

	try {
		JBBPOut joparam = JBBPOut.BeginBin(JBBPByteOrder.valueOf("BIG_ENDIAN"), JBBPBitOrder.valueOf("MSB0"));
		joparam.String("zzzz").Align();
		joparam.Int(10).Align();
		byte[] joparamarray = joparam.End().toByteArray();
		
		System.out.println("the generate byte str is : ");
		System.out.println("--------------------------------");
		System.out.println(JBBPUtils.bin2str(joparamarray, true));
		System.out.println("--------------------------------");
		final JBBPParser tcpParser = JBBPParser.prepare(
				"stringj fin;"
				+ "int cot;"
	      , JBBPBitOrder.valueOf("MSB0"));
		final JBBPFieldStruct bitflds =tcpParser.parse(joparamarray);
		String finstr=bitflds.findFieldForNameAndType("fin", JBBPFieldString.class).getAsString();
		System.out.println("but the parsed str is : ");
		System.out.println("--------------------------------");
		System.out.println(finstr);
		System.out.println(bitflds.findFieldForNameAndType("cot", JBBPFieldInt.class).getAsInt());
		System.out.println("--------------------------------");
		System.out.println("the parsed 'zzzz' binary code is :");
		System.out.println("--------------------------------");
		System.out.println(JBBPUtils.bin2str(finstr.getBytes(),true));
		System.out.println("--------------------------------");
		long endTime = System.currentTimeMillis();
		System.out.println("Full running:" + (endTime - startTime) + "ms");

	} catch (Exception e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	}
}

and the test result is below:

the generate byte str is :

00100000 01111010 01111010 01111010 01111010 00000000 00000000 00000000 01010000

but the parsed str is :

^^^^
10

the parsed 'zzzz' binary code is :

01011110 01011110 01011110 01011110

Full running:98ms

I noticed the binary code of "zzzz" is "01111010 01111010 01111010 01111010", but when I parse use MSB0 , it get the wrong result - '^^^^'.

if I use LSB0 in generater and parser , the result is correct.

but the real request is : the device generate signal in MSB0 bit order.

would you give me some advise to get the correct result ?

thanks a lot !

Hi,

I am getting the following error:

Detected atomic field definition without type

When running the following:

final JBBPParser marketParser = JBBPParser.prepare(
                "byte[5] blob5; "
                + "accountFlags {"
                + " byte[32] address; "
                + "}; "
                + "byte[7] blob7;"
);

I suspect it has to do with the fact that I am passing 32 into the byte array for address. The example:

final JBBPParser pngParser = JBBPParser.prepare(
              "long header;"
              + "chunk [_]{"
              + "   int length; "
              + "   int type; "
              + "   byte[length] data; "
              + "   int crc;"
              + "}"
      );

shows you passing the length which is calculated as the message is read dynamically. Is there a way to statically set this to 32?

Good day,

I am a intermediate level programmer and I have been unable to exactly understand how to use some of the methods within the code base. Partly because of my ignorance to bitwise operations.

Specifically, I am trying to parse a byte block that is 1024 bytes. Essentially, all though not exclusively, I am trying to parse some number of bits for a 4 byte block (32 bits). In some blocks it may be 15 bits, while in another it might be 7 bits.

How can I use the "script" or some other method to accomplish my goal. I just don't quite. understand the notations.

Any help would be really appreciated.

Thanks,

Wyatt

Hi @raydac ,

First of all, thanks for such a brilliant parser, it is very helpful! I am using it to parse TCP packets in Spark streaming.

Currently, i notice JBBP throws parsing exception occasionally when running in multithread, while the same program works well in single thread. And if i copy the failed packet to unit test, it also passes. Would you mind taking a look?

My IP/TCP structure definition is:

"skip:14; // skip bytes till the frame\n"
            +"bit:4 InternetHeaderLength;"
            +"bit:4 Version;"
            +"bit:2 ECN;"
            +"bit:6 DSCP;"
            +"ushort TotalPacketLength;"
            +"ushort Identification;"
            +"bit:8 IPFlagsAndFragmentOffset_low;"
            +"bit:5 IPFlagsAndFragmentOffset_high;"
            +"bit:1 MoreFragment;"
            +"bit:1 DonotFragment;"
            +"bit:1 ReservedBit;"
            +"ubyte TTL;"
            +"ubyte Protocol;"
            +"ushort HeaderChecksum;"
            +"int SourceAddress;"
            +"int DestinationAddress;"
            +"byte [(InternetHeaderLength-5)*4] Options;"

"skip:34; // skip bytes till the frame\n"
            + "ushort SourcePort;"
            + "ushort DestinationPort;"
            + "int SequenceNumber;"
            + "int AcknowledgementNumber;"
            + "bit:1 NONCE;"
            + "bit:3 RESERVED;"
            + "bit:4 HLEN;"
            + "bit:1 FIN;"
            + "bit:1 SYN;"
            + "bit:1 RST;"
            + "bit:1 PSH;"
            + "bit:1 ACK;"
            + "bit:1 URG;"
            + "bit:1 ECNECHO;"
            + "bit:1 CWR;"
            + "ushort WindowSize;"
            + "ushort TCPCheckSum;"
            + "ushort UrgentPointer;"
            + "byte [HLEN*4-20] Option;"
            + "byte [_] Data;"

And the exception i got is:

com.igormaznitsa.jbbp.exceptions.JBBPParsingException: Detected negative calculated array length for field 'options' [-80 (0xFFFFFFB0)]
     at com.igormaznitsa.jbbp.JBBPParser.assertArrayLength(JBBPParser.java:101)
     at com.igormaznitsa.jbbp.JBBPParser.parseStruct(JBBPParser.java:176)
     at com.igormaznitsa.jbbp.JBBPParser.parse(JBBPParser.java:489)
     at com.igormaznitsa.jbbp.JBBPParser.parse(JBBPParser.java:515)

and

JBBPParsingException while parsing, Detected negative calculated array length for field 'option' [-68 (0xFFFFFFBC)]

Thanks in advance!

Hi! Thanks a lot for your work for this repo! Faced a problem with mapping to static field in my project: i need to parse two types of frames( config and data). Config contains info about datatypes and length of binary blocks in the data frames.
So, if i got from the config frame, that all ongoing data frames does not have field X, and there is a field in the mapping class public @Bin (type = BinType.BYTE_ARRAY) byte[] X i'll have "can't find value for mapping field [public byte[] X -> null" . Are there any ways processing this exception, except generating new java template class for each case without nullable field X?

Hi @raydac this library look really cool !

But can i run it with an Android project?

Where is the doc that shows how to unpack? I'm just seeing how to pack

thks