ahausladen / jsondataobjects Goto Github PK
View Code? Open in Web Editor NEWJSON parser for Delphi 2009 and newer
License: MIT License
JSON parser for Delphi 2009 and newer
License: MIT License
When I clone '{ "foo": [ "bar", {}, null, true, false, { "keaa": "value" } ] }', my result at "ClonedObj.ToJSON(True)" is '{"fo":["ba",{},null,true,false,{"ke":"valu"}]}'.
Analyzing, can you see that keys are resulting without the last char.
I'm using the example shown in the Readme .
Amazing work on issue #18! I get a JSON string now in ARM64!
But something is wrong with the string reference count I think. When the JSON string that gets returned goes out of scope an exception is raised. Consider this code:
procedure TForm1.Button1Click(Sender: TObject);
var
jo: TJSONObject;
s: string;
begin
jo := TJSONObject.Create;
jo.S['USERID'] := 'S.TESTER';
s := jo.ToJSON(true); // Get a value now :)
s := ''; // Exception :(
jo.Free;
end;
Exception is also raised if I do not set the string to empty and the method ends (string goes out of scope). This is where the exception is:
function System._UStrClr()
24641: FreeMem(P);
Is the current master branch the beta 1.0 release?
For XE7 I get an compile error: [dcc32 Error] JsonDataObjects.pas(3894): E2036 Variable required
oops forget it my mistake, sorry
Any plans for JsonPath support - I'd be willing to add the support if you haven't already got something planned?
Hi, I had some problems compiling the TBCEditor control, because of a compilation error with XE7 (no updates installed).
See: https://github.com/bonecode/TBCEditor/issues/400
The fix was to remove "overload" from TJsonStringBuilder.AppendIntro (which doesn't have any overloads). Without that change, XE7 doesn't seem to understand that you try to pass a method pointer instead of executing it. It's probably a bug in the compiler, but the fix seems to be easy and I don't think it'll affect other versions.
When trying to use JsonDataObjects we found that on one of ours computers it gives in XE7 (without Update 1) three E2036 Variable required errors. These are on lines 1965, 3755, 3760. This happens only on this one computer (not reproducible on others (XE7 Update 1 installed)) but we are unsure what causes this.
Hi,
The JsonDataObjects can't parse this json string:
[1,2,3,\"This is a test\",{\"a\":\"a\"}]
You can get that json string by executing this code in Firefox or Chrome:
JSON.stringify([1, 2, 3, "This is a test", {"a": "a"}]);
Would be great to have Linux support for JsonDataObjects. In 10.2 Tokyo, ASM sections are not supported on Linux.
TStringIntern.Init contains the following code:
FCount := 0;
FCapacity := 16;
GetMem(FStrings, FCapacity * SizeOf(FStrings[0]));
FCapacity := 17;
GetMem(FBuckets, FCapacity * SizeOf(FBuckets[0]));
FillChar(FBuckets[0], FCapacity * SizeOf(FBuckets[0]), $FF);
This sets the length of FStrings to 1 less than FBuckets. In TStringIntern.InternAdd, TStringInterm.Grow is only triggered when the size reaches FCapacity (now 17):
if FCount = FCapacity then
Grow;
Index := FCount;
Inc(FCount);
Bucket := @FBuckets[(AHash and $7FFFFFFF) mod FCapacity];
with FStrings[Index] do
begin
Next := Bucket^;
Hash := AHash;
Pointer(Name) := Pointer(S);
Inc(PLongint(@PByte(Name)[-8])^);
end;
This means that when FCount reaches 16, writes are performed to FStrings[16], which is beyond the memory allocated during Init.
Hi Andy,
This library is not functioning correctly on iOS 64 bit targets (ARM64). All other targets seem to work fine. I have some sample code at the end of this post.
I have dug a bit and found this so far:
class procedure TJsonBaseObject.StrToJSONStr(const AppendMethod: TWriterAppendMethod; const S: string);
2266: //EndP := P + Length(S);
2267: EndP := P + PLongInt(@PByte(S)[-4])^;
Line 2267 is calculating huge EndP values in ARM64 and causing the while loops to crash. However, if I comment out line 2267 and use line 2266 then EndP correctly points to the line end.
Next:
procedure TJsonOutputWriter.TJsonStringBuilder.DoneConvertToString(var S: string);
6803: Pointer(S) := P; // keep the RefCnt=1
In ARM64 after line 6803 executes S is not updated and still points to the blank text it was initialized to earlier in this method. Line 6803 seems to have no effect when investigated in the debugger.
Sample code:
procedure TForm1.Button1Click(Sender: TObject);
var
jo: TJSONObject;
begin
jo := TJSONObject.Create;
jo.I['ACCOUNTID'] := 225;
jo.B['BOOLVAL'] := False;
jo.S['VERSION'] := '1.0';
jo.S['LOCATION'] := 'RVK';
jo.S['USERID'] := 'S.TESTER';
jo.S['GARBAGE#####'] := 'Ooga\booga';
jo.S['SUBTOCALL'] := 'G2.WEB.CONNECT.SUB';
jo.D['DT'] := Now;
jo.S['ACTION'] := 'CONNECT';
Memo1.Text := jo.ToJSON(true);
// Crash if line 2267 is active, no crash if line 2266 is active.
// When no crash then this always returns an empty string.
jo.Free;
end;
For instance https://github.com/bonecode/TBCEditor/issues/303 uses the objects, but when there is an error in the JSON, the exception is very generic, having a column/line number would help figure out what's wrong.
Minimally this would be for raw JSON syntax errors (like an unbalanced array), and ideally at the DOM level, so that the application that uses the DOM can provide meaningful error (like "field whatever line xxx should be a string")
Following code rises Exception:
procedure TForm1.Button1Click(Sender: TObject);
var
jso: TJsonObject;
begin
jso := TJsonObject.Parse('{"ml_model": "J250'#$A'IMEI"}') as TJsonObject;
jso.Free;
end;
Project Project1.exe raised exception class EJsonParserException with message 'Invalid character in string (1, 19)'.
I checked that string complies with the JSON string specification.
Right now, there are several JSON implementations available. Whats the difference of this among others? Is it speed, scalability or a low footprint?
Looked through the unit tests, looked through the source code. Not seeing anything for TArray support ... this is my time looking at the code, so I may be missing something. Is it supported?
From my benchmark JsonDataObjects is the fastest lib for read write json in delphi, It's awersome! but is poor in documentation.
I'm searching for examples for writing large json (over 50000 nodes, less or more 90 MB) with the best performance.
Eg. is better writing the objects with the full path:
var
Obj: TJsonObject;
i: integer;
begin
Obj := TJsonObject.Create;
for i := 0 to 50000 do begin
Obj['foo']['bar']['baz'].A['array'].Add(i);
end;
Obj.Free;
end;
or write by caching the parent, eg.:
var
Obj: TJsonObject;
aArray: TJsonArray;
i: integer;
begin
Obj := TJsonObject.Create;
aArray := Obj['foo']['bar']['baz'].A['array'];
for i := 0 to 50000 do begin
aArray.Add(i);
end;
Obj.Free;
end;
other trick and tips are welcome!
Hi, actually you can't do the following:
var
v: variant;
begin
v := jsonobject['my name'];
end;
It would be nice to have a new property like:
v := jsonobject.V['my name'];
To return value to a variant with the datatype stored in json datavalue. Of course this should raise an exception if you are trying to cast an object or an array.
It would be useful to to have an overload of TObjectArray.Add procedure to add a variant value to an array using the vartype of the value to determine json datatype.
Actually I do this with 2 procedures, I don't want to change your source code because later is a madness to maintain updates.
With best regards
Christian
[DCC Error] JsonDataObjects.pas(2004): E2089 Invalid typecast
[DCC Error] JsonDataObjects.pas(2009): E2089 Invalid typecast
[DCC Error] JsonDataObjects.pas(2052): E2089 Invalid typecast
[DCC Error] JsonDataObjects.pas(6793): E2037 Declaration of 'Realloc' differs from previous declaration
[DCC Error] JsonDataObjects.pas(6797): E2003 Undeclared identifier: 'Capacity'
[DCC Error] JsonDataObjects.pas(6800): E2003 Undeclared identifier: 'FBytes'
[DCC Error] JsonDataObjects.pas(6803): E2008 Incompatible types
[DCC Error] JsonDataObjects.pas(6816): E2008 Incompatible types
[DCC Error] JsonDataObjects.pas(899): E2065 Unsatisfied forward or external declaration: 'TJsonBytesStream.Realloc'
Does the code currently only work for 32-bit targets? (I was compiling it for iOS)
Can you add Marshalling support for record and arrays too ?
(like in https://github.com/onryldz/x-superobject)
Thank you
I have to serialize and transmit data structures like this:
TFoo = class
Html: WideString;
FileContent: UTF8String;
end;
e.g. The producer may be Internet Explorer or IDE which limits the type/encoding.
It seems that I have to cast them to/from string in both sides. Since the content is pretty large, it would be great if the object model could support these two types natively. It should be also more efficient as I use UTF8-encoding for transmission.
Can you add UInt64 support?
Some real time web-services like Periscope use Net Time Protocol data format. It's UInt64 (explain).
Example:
{
"message": {
"channel": "blablabla",
"ntpForBroadcasterFrame": 14567816253328083,
"ntpForLiveFrame": 14567816253328083,
"participant_index": 1,
"signature": "blablabla",
"signer_str_ntpForBroadcasterFrame": "14567816253328083",
"signer_str_ntpForLiveFrame": "14567816253328083"
},
"channel": "blablabla"
}
Wrong test code:
var
s1, s2: string;
json: TJsonObject;
begin
s1 := '{"long_val":15744383709429629494,"long_str":"15744383709429629494"}';
json := TJsonObject.Parse(s1) as TJsonObject;
s2 := json.ToJSON;
Assert(s1 = s2);
end;
Hi,
Thank you for your great work. Do you have any plan to implement "Object schema validation", similar to https://github.com/hapijs/joi
Thank you
Sometimes values are encoded in Base64. After parsing file there is a need to decode such value. It would be nice if JsonDataObjects library provided function to decode the value.
Sample json:
{
"promotions":[
{
"id":1,
"title":"Special offer",
"start_at":"2014-11-01 00:00:00",
"finish_at":"2014-11-05 00:00:00",
"promotion_type":1,
"promotion_value":5000,
"promotion_on_type":1,
"products":[
1,
2
],
"promotion_on_value":[
1,
2
],
"activate":2,
"created_at":"2011-09-22 01:15:35"
},
{
"id":2,
"title":"Winter sale",
"start_at":"2014-12-01 00:00:00",
"finish_at":"2014-12-30 00:00:00",
"promotion_type":2,
"promotion_value":25,
"promotion_on_type":1,
"products":[
1,
2
],
"promotion_on_value":[
1,
2
],
"activate":2,
"created_at":"2011-09-22 01:15:35"
}
]
}
[DCC Error] JsonDataObjects.pas(1468): E2003 Undeclared identifier: 'TSystemTime'
[DCC Error] JsonDataObjects.pas(1470): E2003 Undeclared identifier: 'DateTimeToSystemTime'
[DCC Error] JsonDataObjects.pas(1472): E2029 ']' expected but identifier 'wYear' found
But for Windows 32 , above errors does not appear.
/*1.json
{"receiver":{"type":"single","id":"lisi"},"sender":"zhangsan","msgtype":"text","text":{"content":"Holiday Request For Pony(http://xxxxx)"}}
*/
TJsonObject *Json=new TJsonObject();
Json->LoadFromFile(L"1.json");
ShowMessage(Json->O[L"text"]->S[L"content"]);
delete Json;
[ilink32 Error] Error: Unresolved external '__fastcall Jsondataobjects::TJsonObject::GetObjectW(System::UnicodeString)' referenced from UNIT1.OBJ
Compile environment : Win10 64bit & C++Builder 10.2.3 Community Edition
In addition, please add {$HPPEMIT ' #pragma link "Jsondataobjects"} to the source code for ease of use of CB users, thank you.
[DCC Error] JsonDataObjects.pas(1386): E2003 Undeclared identifier: 'UtcDateTime'
[{
"2": "[{\"xx\":11,\"a\":1,\"b\":\"000000000\",\"c\":1,\"d\":\"2018-04-23 13:36:18\",\"e\":6,\"f\":\"\",\"g\":12,\"h\":\"00000000\",\"i\":1472,\"j\":\"\",\"k\":\"/8a44f9442e64.jpg\",\"l\":\"553031\",\"m\":\"1\",\"n\":0,\"o\":\"100002000000000\",\"p\":\"86,85,84,33,32,31,\",\"q\":\"fbeeeeee48df429399118\"}]"
}]
In TestJsonDataObjects.pas, in procedure TestTJsonBaseObject.TestVariant, there's a call to CompareFloatRelative().
I have searched the Net and all my code repo (of about 20 years) and I couldn't find it.
It turns out it exists in XE10 (and maybe in other versions above XE2, I didn't check) and since it is MPL licensed, I am copying the relevant code below --in case anyone else needs it.
BTW, these are overloaded.
function CompareFloatRelative(expected, actual: Extended) : Boolean;
begin
Result := CompareFloatRelative(expected, actual, 0.0000001); //This is a 99.99999% accuracy
end;
function CompareFloatRelative(expected, actual: Extended; RelativeError : Extended) : Boolean;
begin
Result := False;
if expected = actual then := True
else if abs((actual - expected) / expected) < RelativeError then Result := True;
end;
Hello,
The SaveFromStream works, but LoadFromStream raise an exception.
XE3
JsonDataObjects.pas(2117) Error: E2003 Undeclared identifier: 'TryStrToUInt64'
Accidentally stumbled upon this:
JsonDataObjects/Source/JsonDataObjects.pas
Line 1156 in 72a25ad
Should that line not use JsonMemInfoMainBlockStart
instead of JsonMemInfoBlockStart
?
There seems to be some support for auto ref counting but when i define AUTOREFCOUNT can't compile project in D2009
[DCC Error] JsonDataObjects.pas(2257): E2003 Undeclared identifier: 'FRefCount'
[DCC Error] JsonDataObjects.pas(2270): E2003 Undeclared identifier: 'FRefCount'
[DCC Error] JsonDataObjects.pas(2777): E2003 Undeclared identifier: '__ObjRelease'
[DCC Error] JsonDataObjects.pas(2783): E2003 Undeclared identifier: 'AtomicIncrement'
[DCC Error] JsonDataObjects.pas(2783): E2003 Undeclared identifier: 'FRefCount'
[DCC Error] JsonDataObjects.pas(4677): E2003 Undeclared identifier: 'FRefCount'
[DCC Error] JsonDataObjects.pas(4690): E2003 Undeclared identifier: 'FRefCount'
It is often not necessary to create DOM (JsonObject) when serializing objects to a stream. It is more efficient to expose a public TJsonWriter to perform such task.
TJsonOutputWriter is used internally so far. Also, UTF8String should be considered in this part.
Sometimes the date given to JsonDataObjects is in UTC already, for example when generated by functions such as EncodeDate(). When a local time offset is then subtracted internally by JsonDataObjects you can end up with a day earlier and some hours offset.
Maybe another property additionally to D[], such as DUtc[] would work, so you can intentionally set some dates as UTC and some as local time/date.
TJsonSerializationConfig.UseUtcTime is not enough, since TzSpecificLocalTimeToSystemTime() will be called regardless of that setting.
Thanks for the very well designed interface, btw!
I'm using JsonDataObjects to serialize DTO to send to Java REST Server, but the Java serializer does not recognize Names with UpperCaseStartingCamelCase, it must be lowercaseStartingCamelCase, and I cannot change the property names in Delphi Objects, I have to follow the Delphi Naming convention.
To workaround this problem, I changed TJsonObject.FromSimpleObject, to convert property name
if StartNamesLowerCase then
PropName := LowerCase(UTF8ToString(PropList[Index].Name).Chars[0]) + UTF8ToString(PropList[Index].Name).Substring(1)
else
PropName := UTF8ToString(PropList[Index].Name);
Is possible to have this feature built in?
Hi Andreas,
I suggest to change this line:
https://github.com/ahausladen/JsonDataObjects/blob/master/Source/JsonDataObjects.pas#L2571
From:
Result := string(FValue.S) = 'true';
to:
Result := BoolToStr(string(FValue.S));
... this would allow to use "True" or "TRUE" also inside the JSON file...
now if we try to cast a null value to any type of data an exception is raised:
var
doc: TJsonObject;
v: variant;
begin
doc := TJsonObject.Create;
try
doc['nullvalue'] := null;
// any of this raise an exception
v := doc['nullvalue'].Value;
v := doc['nullvalue'].IntValue;
v := doc['nullvalue'].LongValue;
v := doc['nullvalue'].FloatValue;
v := doc['nullvalue'].DateTimeValue;
v := doc['nullvalue'].BoolValue;
finally
doc.Free;
end;
end;
Why don't we cast nulls like if it is jdtNone?
Also DataSet fields cast nulls to required datatypes:
// if DataField.IsNull
v := DataField.AsString; // this return empty string
v := DataField.AsInteger; // this return 0
...
Also VarToString function:
v := VarToString(null); // this return empty string
I would like to implement this but I like to ask you first.
Regards
Christian
Can Delphi 2007 be supported,Downward compatibility with lower versions!
Delphi Rio 10.3.2
TJsonObject.FromSimpleObject use GetPropList. But GetPropList works with only published.
Directive {$M+} doesn't work.
Help says GetPropList only works for components or component type.
Hi, there is a bug when using .path to set values.
I can it reproduce with following code:
procedure TForm2.Button2Click(Sender: TObject);
var
doc: TJsonObject;
v: variant;
begin
doc := TJsonObject.Create;
doc.Path['ferrcod'] := 2;
doc.Path['ferrmsg'] := 'Debe haber un usuario';
Memo1.Lines.Text := doc.ToJSON(false);
end;
Instead of returning:
{
"ferrcod": 2
"ferrmsg": "Debe haber un usuario"
}
It returns:
{
"ferrcod": "Debe haber un usuario"
}
I'm trying the find the problem but I still not found.
I'm working on a package manager, (currently called DPM, might change), I create a pull request to add dpm support for jsondataobjects - it just adds the a dspec file which defines what goes in the package.
Can I ask that you also add "dpmpackage" as a topic to the repo (manage topics under the repo name). That will make it possible to find it when searching for dpm packages.
FWIW, I'm using jsondataobjects in DPM itself.
Hi!
I have a 100Mb Json file, it takes a while to read. It is possible to show a progress state?
Thanks.
add feature, configure json serialization engine with option ignore fields with empty value
I'm interested in implement a jdtDateTime type to handle of DateTime values in the same way that Float values are handled.
This will let us to transparently handle DateTime values instead of manually convert or format them when needed.
This also save memory because it will store a double value instead of 24 chars.
Some examples of use could be:
procedure TForm2.Button2Click(Sender: TObject);
var
doc: TJsonObject;
variantvar: variant;
datetimevar: TDateTime;
stringvar: string;
begin
doc := TJsonObject.Create;
try
// assign a datetime value and internally store as Double, with Typ = jdtDateTime
doc['mydatetimevalue'] := now;
doc.D['mydatetimevalue'] := now;
// return value as TDateType
variantvar := doc['mydatetimevalue'];
variantvar := doc['mydatetimevalue'].DateTimeValue;
variantvar := doc.D['mydatetimevalue'];
// return value as TDateType
datetimevar := doc['mydatetimevalue'];
datetimevar := doc['mydatetimevalue'].DateTimeValue;
datetimevar := doc.D['mydatetimevalue'];
// cast to string to standard JsonFormat
stringvar := doc['mydatetimevalue'];
stringvar := doc['mydatetimevalue'].Value;
// output dates as standard JSON format
doc.ToJSON; // will output -> {"mydatetimevalue": "yyyy-mm-ddThh:mm:ss.zzzZ"}
finally
doc.Free;
end;
end;
I think that this doesn't broke or change anything. But I want to hear your opinions before start to work.
If you agree I will do it in my fork and then I'll put a pull request.
How to get each item(Object, not a single Value) from the ARRAY?
thks!
Check the position of semicolon in function parameters
Parsing seems to fail for all Int64 values x
where MaxInt < x <= 2*MaxInt + 1
. Other values work fine. The problem seems to be that FLook.HI = 0
and therefore FLook.I
is used instead of FLook.L
. Unfortunately I don't know how to fix it so I can't provide you with a pull request.
var
lInput: String;
lInt64: Int64;
lJsonObject: TJsonObject;
begin
lInt64 := 2 * Int64(MaxInt);
lInput := Format('{ "num": %d }', [lInt64]);
lJsonObject := TJsonObject.Parse(lInput) as TJsonObject;
try
Writeln(lInput); // { "num": 4294967294 }
Writeln(lInt64); // 4294967294
Writeln(lJsonObject.L['num']); // -2
finally
lJsonObject.Free;
end;
end;
This is Delphi XE5.
Hi, when you parse a Json string, null values are stored as jdtObject instead of jdtNone.
Can be reproduced with following code:
procedure TForm2.Button2Click(Sender: TObject);
var
doc: TJsonObject;
begin
doc := TJsonObject.Create;
try
doc.FromJSON('{"ferrcod": 2, "ferrmsg": null, "ferrval": "pepe"}');
if doc['ferrmsg'].Typ <> jdtNone then
raise Exception.Create('Why???');
finally
doc.Free;
end;
end;
This doesn't happen when values are assigned one by one:
procedure TForm2.Button2Click(Sender: TObject);
var
doc: TJsonObject;
begin
doc := TJsonObject.Create;
try
doc['ferrcod'] := 2;
doc['ferrmsg'] := null;
doc['ferrval'] := "pepe";
if doc['ferrmsg'].Typ = jdtNone then
raise Exception.Create('Now its ok');
finally
doc.Free;
end;
end;
When I use JsonDataObjects,pas (in Delphi XE) like this:
procedure doParseUTF8();
var
aJson: TJsonObject;
aJsonStr: string;
aUTF8Str: UTF8String;
begin
aJsonStr := '{"123":{"abc":{"efg":"hij"}}}';
aUTF8Str := UTF8Encode(aJsonStr);
aJson := TJsonObject.Create;
try
aJson.FromUtf8JSON(aUTF8Str);
ShowMessage(aJson.ToJSON);
finally
aJson.Free;
end;
end;
I get message '{"12":{"ab":{"ef":"hi"}}}'.
The Key and The Value all have been cut.
this Error also in parse from UTF8 file.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.