sebastianconcept / mapless Goto Github PK
View Code? Open in Web Editor NEWSchema-less persistence for Smalltalk with support for multiple backends.
Home Page: http://sebastianconcept.github.io/Mapless/
License: MIT License
Schema-less persistence for Smalltalk with support for multiple backends.
Home Page: http://sebastianconcept.github.io/Mapless/
License: MIT License
This bug can be reproduced using considerable concurrent load.
I was able to reproduce it consistently with 30 concurrent connections hitting a RESTful endpoint that was using the same Mapless repo
MaplessMemoryRepository>>insert: aMapless
self onBeforeInsert: aMapless.
self upsert: aMapless.
self onAfterInsert: aMapless
Should ensure that mapless is saved with an ID so later can be updated with a save:
without duplication.
It has been noticed in production that requests made to the Mongo secondaries of a replica set aren’t balanced, one receiving about 5 times more requests than the other.
There is a case where this exception is improperly signalled.
MaplessMongoRepository(MaplessRepository)>>asStorable: anObject
^ anObject isCollection
ifTrue: [ anObject class = OrderedJsonObject
ifTrue: [ (JsonObject fromAssociations: anObject associations)
collect: [ :e | self resolver maplessAsStorable: e in: self ] ]
ifFalse: [ anObject collect: [ :e | self asStorable: e ] ] ]
ifFalse: [ anObject class = MaplessReference
ifFalse: [ (anObject isKindOf: Mapless)
ifTrue: [ (resolver asMaplessReferenceIn: anObject in: self)
asJsonObjectIn: self ]
ifFalse: [ anObject ] ]
ifTrue: [
anObject hasModel
ifTrue: [ resolver maplessReferenceAsJsonObject: anObject in: self ]
ifFalse: [ MaplessUnsavedSubmodel
signal:
'This sub model is unsaved. You need to save all sub models before saving a composed model' ] ] ]
Noting the part:
...
ifTrue: [
anObject hasModel
ifTrue: [ resolver maplessReferenceAsJsonObject: anObject in: self ]
ifFalse: [ MaplessUnsavedSubmodel
...
The case happen when the submapless was indeed saved but it was not unreferenced, so it will have no model but not because wan't saved, it was only because it was not used yet, hence the reference not unreferenced.
Right now Mapless uses jQuery ajax calls. This makes development a little tricky when amber node server and the backend are running on different IPs.
Currently a MaplessReference
will instantiate the referenced object, regardless of whether such reference was instatiated before in the same repository, producing multiple instances of the same "logical" object.
A MaplessResference
should have Resolver
(reference resolver), probably defined in the MaplessRepository
, then the reference will go through the resolver, the default resolver will preserve the current behavior to provide backwards compatibility (and simplicity), but we could introduce a CacheResolver
that will first look into a cache before going to the database to fetch the object from the collection.
provided implemented changes in a comment as I did not manage to overcome access rights
If a Mapless class has children, when querying the parent, should bring as results, the corresponding instances of these children classes.
When a MaplessMongoError is signalled, it should contain native details about the cause when possible.
This would allow apps to handle it reflecting on the MaplessMongoError code for example.
When the MaplessMongoRepository
is using as accessor a MaplessMongoReplicaSetPool
, the read-only and read-write operations should be performed gracefully when changes in the MongoDB nodes are happening.
For making applications unit test autonomous and maybe other uses, a memory only repository for Mapless will be useful.
In order to apply MongoDB cluster load balance for reads and writes, the Mapless connection pool needs to be improved so it can have separated sets of connections:
Make Mapless API for a MongoDB repository, able to use MQQuery.
Provide a logging mechanism to make a database to be auditable from a central source.
MongoDB is a must.
Factorized to make it easier for other backends is a plus.
Loading 5.5 as a dependency from a BaselineOf shows an issue with the load order of some Mapless test classes
Answer this:
How interesting is MCDataStream serialization compared for example with BSON?
reified := (MCDataStream on: bytes contentStream) next.
There are several optimization opportunities in this critical methods that get a client from the pool.
As a starter, getting a read-only client is way slower than getting a read-write client:
ABBench bench: [
ABBench
a: [ maplessPool getIdleReadWriteClient ]
b: [ maplessPool getIdleReadOnlyClient ]
].
"B is 99.19% SLOWER than A"
Below, a profiler for reference:
Reporting - 44,416 tallies, 5,021 msec.
**Tree**
100.0 (5,021) BlockClosure newProcess
100.0 (5,021) MorphicUIManager spawnNewProcess
100.0 (5,021) WorldMorph class doOneCycle
100.0 (5,021) WorldMorph doOneCycle
100.0 (5,021) WorldState doOneCycleFor:
100.0 (5,021) WorldState doOneCycleNowFor:
100.0 (5,021) WorldMorph runStepMethods
100.0 (5,021) WorldState runStepMethodsIn:
100.0 (5,021) WorldState runLocalStepMethodsIn:
100.0 (5,021) WorldState triggerAlarmsBefore:
100.0 (5,021) MorphicAlarm value:
100.0 (5,021) MorphicAlarm [MessageSend] value
100.0 (5,021) GLMMorphicPharoScriptRenderer [GLMMorphicPharoCodeRenderer] popupPrint
100.0 (5,021) RubSmalltalkEditor highlightEvaluateAndDo:
100.0 (5,021) RubSmalltalkEditor evaluate:andDo:
100.0 (5,021) OpalCompiler evaluate
100.0 (5,021) UndefinedObject DoIt
100.0 (5,021) BlockClosure bench
100.0 (5,021) BlockClosure benchFor:
100.0 (5,021) UndefinedObject DoIt
100.0 (5,021) MaplessMongoReplicaSetPool readOnlyDo:
99.9 (5,016) BlockClosure on:do:
99.80000000000001 (5,011) MaplessMongoReplicaSetPool readOnlyDo:
99.80000000000001 (5,011) BlockClosure on:do:
99.60000000000001 (5,001) MaplessMongoReplicaSetPool readOnlyDo:
94.5 (4,745) MaplessMongoReplicaSetPool requestReadOnlyClient
|94.5 (4,745) Mutex critical:
| 94.4 (4,740) Semaphore critical:
| 94.30000000000001 (4,735) BlockClosure ensure:
| 94.30000000000001 (4,735) Semaphore critical:
| 94.30000000000001 (4,735) Mutex critical:
| 92.60000000000001 (4,649) BlockClosure ensure:
| 92.60000000000001 (4,649) MaplessMongoReplicaSetPool requestReadOnlyClient
| 90.7 (4,554) MaplessMongoReplicaSetPool getIdleReadOnlyClient
| |45.6 (2,290) IdentitySet [Set] collect:
[45.400000000000006 (2,280) Array [SequenceableCollection] do:
[ 45.300000000000004 (2,275) IdentitySet [Set] collect:
[ 45.300000000000004 (2,275) MaplessMongoReplicaSetPool getIdleReadOnlyClient
[ 45.300000000000004 (2,275) MongoAPI asMongoUrl
[ 45.2 (2,269) ByteString [String] asMongoUrl
[ 45.2 (2,269) ZnUrl class fromString:defaultScheme:
[ 45.2 (2,269) ZnUrl parseFrom:defaultScheme:
[ 18.3 (919) ZnUrl parseAuthority:from:to:
[ |15.5 (778) ZnUrl parseHostPort:
[ | |12.3 (618) ReadStream [PositionableStream] atEnd
[ | | |0.6000000000000001 (30) ByteString class translate:from:to:table:
[ | |1.7000000000000002 (85) ReadStream upTo:
[ | | |1.5 (75) ByteString [SequenceableCollection] copyFrom:to:
[ | | | |0.4 (20) ByteString class indexOfAscii:inString:startingAt:
[ | | |0.2 (10) ByteString [String] indexOf:startingAt:ifAbsent:
[ | |1.0 (50) ZnUrl decodePercent:
[ | | |1.0 (50) ZnResourceMetaUtils class decodePercent:
[ | | | 1.0 (50) ZnPercentEncoder decode:
[ | | | 0.5 (25) ZnUTF8Encoder [ZnUTFEncoder] decodeBytes:
[ | | | |0.5 (25) String class [SequenceableCollection class] streamContents:
[ | | | | 0.2 (10) String class [SequenceableCollection class] new:streamContents:
[ | | | | 0.2 (10) ZnUTF8Encoder [ZnUTFEncoder] decodeBytes:
[ | | | | 0.2 (10) ZnUTF8Encoder nextCodePointFromStream:
[ | | | | 0.2 (10) ReadStream next
[ | | | 0.30000000000000004 (15) ByteArray class [SequenceableCollection class] streamContents:
[ | | | |0.1 (5) ByteArray class [SequenceableCollection class] new:streamContents:
[ | | | | 0.1 (5) WriteStream class [PositionableStream class] on:
[ | | | 0.1 (5) ByteString [SequenceableCollection] readStream
[ | |0.4 (20) Integer class readFrom:ifFail:
[ |1.0 (50) ReadStream class [PositionableStream class] on:from:to:
[ | |0.5 (25) ByteString class indexOfAscii:inString:startingAt:
[ |0.5 (25) ByteString class indexOfAscii:inString:startingAt:
[ 12.8 (643) ByteString [String] indexOf:
[ |0.7000000000000001 (35) String findSubstringViaPrimitive:in:startingAt:matchTable:
[ |0.5 (25) ByteString class indexOfAscii:inString:startingAt:
[ 5.2 (261) ZnUrl isSchemeNotUsingDoubleSlash:
[ |5.2 (261) Array [SequenceableCollection] includes:
[ | 3.4000000000000004 (171) Array [SequenceableCollection] indexOf:
[ | |3.4000000000000004 (171) Array [SequenceableCollection] indexOf:ifAbsent:
[ | | 3.4000000000000004 (171) Array [SequenceableCollection] indexOf:startingAt:ifAbsent:
[ | | 1.2000000000000002 (60) ByteString class compare:with:collated:
[ | 0.6000000000000001 (30) ByteString class translate:from:to:table:
[ 3.7 (186) ZnUrl scheme:
[ |3.7 (186) ByteString [String] asSymbol
[ | 1.3 (65) Symbol class intern:
[ | |1.3 (65) Symbol class lookup:
[ | | 1.3 (65) WeakSet like:
[ | | 1.3 (65) WeakSet scanFor:
[ | | 1.3 (65) ByteSymbol [Symbol] =
[ | | 0.4 (20) ByteString class stringHash:initialHash:
[ | 0.6000000000000001 (30) ByteString class translate:from:to:table:
[ 2.4000000000000004 (121) ZnUrl hasScheme
[ |0.6000000000000001 (30) ByteString class compare:with:collated:
[ 1.4000000000000001 (70) ByteString [SequenceableCollection] copyFrom:to:
[ |0.5 (25) ByteString class indexOfAscii:inString:startingAt:
[ 1.4000000000000001 (70) ByteString [String] indexOfSubCollection:
[ 0.5 (25) ByteString class indexOfAscii:inString:startingAt:
[ 0.5 (25) ByteString [String] indexOfSubCollection:startingAt:ifAbsent:
[ 0.2 (10) ByteString [String] findString:startingAt:
[ 0.2 (10) ByteString [String] findString:startingAt:caseSensitive:
[ 0.2 (10) ByteString class [Behavior] isBytes
[ 0.2 (10) ByteString class [Behavior] instSpec
| |33.300000000000004 (1,672) IdentitySet [Collection] detect:ifNone:
[33.300000000000004 (1,672) IdentitySet [Collection] detect:ifFound:ifNone:
[ 33.2 (1,667) IdentitySet [Set] do:
[ 31.0 (1,557) IdentitySet [Collection] detect:ifFound:ifNone:
[ |31.0 (1,557) MaplessMongoReplicaSetPool getIdleReadOnlyClient
[ | 28.5 (1,431) MongoAPI asMongoUrl
[ | |28.5 (1,431) ByteString [String] asMongoUrl
[ | | 28.5 (1,431) ZnUrl class fromString:defaultScheme:
[ | | 28.5 (1,431) ZnUrl parseFrom:defaultScheme:
[ | | 12.600000000000001 (633) ZnUrl parseAuthority:from:to:
[ | | |10.700000000000001 (537) ZnUrl parseHostPort:
[ | | | |8.4 (422) ReadStream [PositionableStream] atEnd
[ | | | | |0.4 (20) ByteString class translate:from:to:table:
[ | | | |1.2000000000000002 (60) ReadStream upTo:
[ | | | | |1.0 (50) ByteString [SequenceableCollection] copyFrom:to:
[ | | | | | |0.30000000000000004 (15) ByteString class indexOfAscii:inString:startingAt:
[ | | | | |0.30000000000000004 (15) ByteString [String] indexOf:startingAt:ifAbsent:
[ | | | |0.8 (40) ZnUrl decodePercent:
[ | | | | |0.8 (40) ZnResourceMetaUtils class decodePercent:
[ | | | | | 0.8 (40) ZnPercentEncoder decode:
[ | | | | | 0.5 (25) ZnUTF8Encoder [ZnUTFEncoder] decodeBytes:
[ | | | | | |0.5 (25) String class [SequenceableCollection class] streamContents:
[ | | | | | | 0.2 (10) String class [SequenceableCollection class] new:streamContents:
[ | | | | | | 0.1 (5) ZnUTF8Encoder [ZnUTFEncoder] decodeBytes:
[ | | | | | | 0.1 (5) ZnUTF8Encoder nextCodePointFromStream:
[ | | | | | | 0.1 (5) ReadStream next
[ | | | | | 0.30000000000000004 (15) ByteArray class [SequenceableCollection class] streamContents:
[ | | | | | 0.1 (5) ByteArray class [SequenceableCollection class] new:streamContents:
[ | | | | | 0.1 (5) WriteStream class [PositionableStream class] on:
[ | | | |0.2 (10) Integer class readFrom:ifFail:
[ | | |0.7000000000000001 (35) ReadStream class [PositionableStream class] on:from:to:
[ | | | |0.30000000000000004 (15) ByteString class indexOfAscii:inString:startingAt:
[ | | |0.30000000000000004 (15) ByteString class indexOfAscii:inString:startingAt:
[ | | 6.4 (321) ByteString [String] indexOf:
[ | | |0.5 (25) String findSubstringViaPrimitive:in:startingAt:matchTable:
[ | | |0.4 (20) ByteString class indexOfAscii:inString:startingAt:
[ | | 3.6 (181) ZnUrl isSchemeNotUsingDoubleSlash:
[ | | |3.6 (181) Array [SequenceableCollection] includes:
[ | | | 2.4000000000000004 (121) Array [SequenceableCollection] indexOf:
[ | | | |2.4000000000000004 (121) Array [SequenceableCollection] indexOf:ifAbsent:
[ | | | | 2.4000000000000004 (121) Array [SequenceableCollection] indexOf:startingAt:ifAbsent:
[ | | | | 0.8 (40) ByteString class compare:with:collated:
[ | | | 0.4 (20) ByteString class translate:from:to:table:
[ | | 2.5 (126) ZnUrl scheme:
[ | | |2.4000000000000004 (121) ByteString [String] asSymbol
[ | | | 0.9 (45) Symbol class intern:
[ | | | |0.9 (45) Symbol class lookup:
[ | | | | 0.9 (45) WeakSet like:
[ | | | | 0.9 (45) WeakSet scanFor:
[ | | | | 0.9 (45) ByteSymbol [Symbol] =
[ | | | | 0.30000000000000004 (15) ByteString class stringHash:initialHash:
[ | | | 0.4 (20) ByteString class translate:from:to:table:
[ | | 1.6 (80) ZnUrl hasScheme
[ | | |0.4 (20) ByteString class compare:with:collated:
[ | | 0.9 (45) ByteString [SequenceableCollection] copyFrom:to:
[ | | |0.30000000000000004 (15) ByteString class indexOfAscii:inString:startingAt:
[ | | 0.8 (40) ByteString [String] indexOfSubCollection:
[ | | 0.30000000000000004 (15) ByteString class indexOfAscii:inString:startingAt:
[ | | 0.2 (10) ByteString [String] indexOfSubCollection:startingAt:ifAbsent:
[ | 2.5 (126) ZnUrl =
[ | 2.5 (126) ZnUrl equals:
[ | 2.5 (126) ZnUrl portOrDefault
[ | 0.4 (20) ZnUrl portIfAbsent:
[ | |0.1 (5) True ifTrue:ifFalse:
[ | 0.2 (10) ByteString class compare:with:collated:
[ 0.2 (10) ByteString class compare:with:collated:
| |6.2 (311) Set class newFrom:
[6.2 (311) Set [Collection] addAll:
[ 6.2 (311) IdentitySet [Set] do:
[ 6.1000000000000005 (306) Set [Collection] addAll:
[ 6.1000000000000005 (306) Set add:
[ 4.2 (211) Set [HashedCollection] atNewIndex:put:
[ |0.2 (10) ByteString class stringHash:initialHash:
[ 1.9000000000000001 (95) Set scanFor:
[ 1.9000000000000001 (95) ZnUrl =
[ 1.1 (55) ZnUrl equals:
[ |1.1 (55) ZnUrl portOrDefault
[ | 0.4 (20) ZnUrl portIfAbsent:
[ | |0.2 (10) True ifTrue:ifFalse:
[ | 0.2 (10) ByteString class compare:with:collated:
[ 0.1 (5) ByteString class stringHash:initialHash:
| |5.6000000000000005 (281) MaplessWeightedRandomPolicy nextAmong:
[3.0 (151) MaplessWeightedRandomPolicy next
[ |1.4000000000000001 (70) SmallInteger *
[ | |0.1 (5) SmallFloat64 truncated
[ |0.6000000000000001 (30) MaplessWeightedRandomPolicy getNext:
[ | 0.6000000000000001 (30) OrderedCollection [Collection] detect:ifNone:
[ | 0.4 (20) OrderedCollection [Collection] detect:ifFound:ifNone:
[ | 0.2 (10) OrderedCollection do:
[2.6 (131) Set includes:
[ 2.6 (131) Set [HashedCollection] findElementOrNil:
[ 2.6 (131) Set scanFor:
[ 2.6 (131) ZnUrl =
[ 1.0 (50) ZnUrl equals:
[ |1.0 (50) ZnUrl portOrDefault
[ | 0.4 (20) ZnUrl portIfAbsent:
[ | |0.2 (10) True ifTrue:ifFalse:
[ | 0.2 (10) ByteString class compare:with:collated:
[ 0.1 (5) ByteString class stringHash:initialHash:
| 1.9000000000000001 (95) MaplessMongoReplicaSetPool removeReadOnlyClient:ifAbsent:
| 0.2 (10) Socket primSocketConnectionStatus:
5.2 (261) MongoCurrentClient class [DynamicVariable class] value:during:
3.6 (181) MongoCurrentClient [DynamicVariable] value:during:
3.6 (181) BlockClosure ensure:
3.6 (181) MongoCurrentClient [DynamicVariable] value:during:
2.8000000000000003 (141) MaplessMongoReplicaSetPool readOnlyDo:
2.8000000000000003 (141) BlockClosure ensure:
2.8000000000000003 (141) MaplessMongoReplicaSetPool readOnlyDo:
2.8000000000000003 (141) MaplessMongoReplicaSetPool returnReadOnlyClient:
2.8000000000000003 (141) Mutex critical:
2.7 (136) Semaphore critical:
[2.6 (131) BlockClosure ensure:
[ 2.6 (131) Semaphore critical:
[ 2.6 (131) Mutex critical:
[ 0.9 (45) BlockClosure ensure:
[ 0.9 (45) MaplessMongoReplicaSetPool returnReadOnlyClient:
[ 0.8 (40) MaplessMongoReplicaSetPool idleReadOnlyClients
[ 0.2 (10) Socket primSocketConnectionStatus:
**Leaves**
4.800000000000001 (241) ByteString class indexOfAscii:inString:startingAt:
3.8000000000000003 (191) ByteString class compare:with:collated:
2.9000000000000004 (146) ByteString class translate:from:to:table:
1.2000000000000002 (60) String findSubstringViaPrimitive:in:startingAt:matchTable:
1.2000000000000002 (60) ByteString class stringHash:initialHash:
0.5 (25) True ifTrue:ifFalse:
0.5 (25) ByteString [String] indexOf:startingAt:ifAbsent:
0.30000000000000004 (15) Socket primSocketConnectionStatus:
0.30000000000000004 (15) ReadStream next
0.2 (10) OrderedCollection do:
0.2 (10) ByteString class [Behavior] instSpec
0.2 (10) ReadStream [PositionableStream] on:
0.1 (5) SmallFloat64 truncated
0.1 (5) WriteStream on:
0.1 (5) Semaphore signal
0.1 (5) NumberParser on:
**Memory**
old +0 bytes
young -1,080,016 bytes
used -1,080,016 bytes
free +1,080,016 bytes
**GCs**
full 0 totalling 0ms (0.0% uptime)
incr 690 totalling 362ms (7.0% uptime), avg 1.0ms
tenures 0
root table 0 overflows
**Processes**
Total process switches: 88849
Without Profiler: 17
Stack page overflows: 875256
Stack page divorces: 35
Make find:
and findOne:
consistent and rename instanceOf:
methods
We have noticed that when a Mongo node becomes unavailable, Mapless does not attempt to reconnect to it.
This can result in serious consequences if multiple nodes become unavailable in succession.
We should provide a simple example where knowing the host:port, database, user and password we can get a MaplessRepository up & running to save our objects into.
We can default to Mongo as sample repository until we polish other backends.
Make the Mapless source code as agnostic and as compatible as possible.
This should be a foundation for porting Mapless to other Smalltalk distributions.
Not all Mapless use cases would find convenient the use of the MaplessCurrentRepository
DynamicVariable.
There are applications that might use Mapless from different repos or do joints from different repos. Mapless has the chance to be a great fit for manipulating data like that.
Implement https://github.com/sebastianconcept/GoHashMap as backend.
We need the count feature.
Make Mapless able to save/load using Redis as persistence backend.
Basically renaming maplessClass
to _maplessClass
MaplessMongoRepository>>findAll: was not using the class metadata used in the find
filter.
This created the problem of returning documents that would belong to different classes that are stored in a common collection.
This would fix it:
findAll: aMaplessClass
^ self
findAll: aMaplessClass
where: (resolver conditionForClass: aMaplessClass)
MaplessResolver>>conditionForClass: aMaplessClass
^ {('_c' -> aMaplessClass name)} asDictionary
MaplessVoyageWithMaplessSuffixResolver>>conditionForClass: aMaplessClass
^ {('#instanceOf' -> (self voyageClassNameFrom: aMaplessClass))}
asDictionary
Currently, when a mapless tries to get its storable form, there are some types that are not supported but might be treated as supported (i.e.: they answer to isCollection
).
Some stress tests revealed poor performance while using upsert in a release candidate application for production.
The plan is to keep the upsert
feature but favor a design using insert
and update
for saves instead of upsert
and re-test performance.
When the db accessor starts, it should ensure a minimum of clients for the pool as configured either during instantiation or with a default of 2.
We have a client available for EJDB https://github.com/pharo-nosql/pharo-ejdb
Mongo-Log format adjusted and some minor conveniences related to it
When the pool gets a client returned, sending an isPrimary
message resulted to be a check that isn't "cheap". As this could affect capacity, this check will be removed.
The health-checking mechanism proved to help the resolver to have the rights clients after MongoDB replica set topology changes in the next check anyway.
The MongoChange
output needs to include:
createdDate
with a Date object showing just the date of the changed document.createdAt
with a Date and TimeStamp of the creation of the changed document.type
of change as uppercaseMongoDB has different options to use when querying in a replica set: ReadConcerns.
Extend Mapless API to allow using that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.