Comments (4)
I would only comment on the code you posted,
var array = CustomSongsDataList.ToArray();
Make sure this is an array of ExtraSongSerializedData.
If possible, could you provide a minimal sample that works?
(The above code is not executable, and the type of CustomSongsDataList is unknown.
That way we may be able to investigate a bit further.
from memorypack.
What I currently use is this, which also produces the same error
using System.Collections.Concurrent;
using System.ComponentModel;
using System.Diagnostics;
using MemoryPack;
using MemoryPack.Streaming;
using Newtonsoft.Json;
namespace ConsoleApp
{
internal class Program
{
private static ConcurrentDictionary<string, ExtraSongData> CustomSongsData = new();
private static string DataPath = "SongCoreExtraData.json";
private static async Task Main(string[] args)
{
var json = await File.ReadAllTextAsync(DataPath);
CustomSongsData = JsonConvert.DeserializeObject<ConcurrentDictionary<string, ExtraSongData>>(json)!;
await MemoryPack();
}
private static async Task MemoryPack()
{
var startTime = Stopwatch.GetTimestamp();
await using var fileStream = File.Open("data.json", FileMode.Create);
await MemoryPackStreamingSerializer.SerializeAsync(fileStream, CustomSongsData.Count, CustomSongsData);
Console.WriteLine($"Serialization: {CustomSongsData.Count} entries in {Stopwatch.GetElapsedTime(startTime)}");
fileStream.Position = 0;
startTime = Stopwatch.GetTimestamp();
CustomSongsData.Clear();
await foreach (var pair in MemoryPackStreamingSerializer.DeserializeAsync<KeyValuePair<string, ExtraSongData>>(fileStream))
{
CustomSongsData.TryAdd(pair.Key, pair.Value);
}
Console.WriteLine($"Deserialization: {CustomSongsData.Count} entries in {Stopwatch.GetElapsedTime(startTime)}");
}
}
[MemoryPackable]
public partial class ExtraSongData
{
public string[] _genreTags;
public Contributor[] contributors;
public string _customEnvironmentName;
public string _customEnvironmentHash;
public DifficultyData[] _difficulties;
public string _defaultCharacteristic = null;
public ColorScheme[] _colorSchemes;
public string[] _environmentNames;
public CharacteristicDetails[] _characteristicDetails;
}
[MemoryPackable]
public partial class MapColor
{
public float r;
public float g;
public float b;
[DefaultValue(1)]
public float a = 1f;
}
[MemoryPackable]
public partial class Contributor
{
public string _role;
public string _name;
public string _iconPath;
}
[MemoryPackable]
public partial class ColorScheme
{
public bool useOverride;
public string colorSchemeId;
public MapColor? saberAColor;
public MapColor? saberBColor;
public MapColor? environmentColor0;
public MapColor? environmentColor1;
public MapColor? obstaclesColor;
public MapColor? environmentColor0Boost;
public MapColor? environmentColor1Boost;
public MapColor? environmentColorW;
public MapColor? environmentColorWBoost;
}
[MemoryPackable]
public partial class RequirementData
{
public string[] _requirements;
public string[] _suggestions;
public string[] _warnings;
public string[] _information;
}
[MemoryPackable]
public partial class CharacteristicDetails
{
public string _beatmapCharacteristicName;
public string? _characteristicLabel;
public string? _characteristicIconFilePath = null;
}
[MemoryPackable]
public partial class DifficultyData
{
public string _beatmapCharacteristicName;
public BeatmapDifficulty _difficulty;
public string _difficultyLabel;
public RequirementData additionalDifficultyData;
public MapColor? _colorLeft;
public MapColor? _colorRight;
public MapColor? _envColorLeft;
public MapColor? _envColorRight;
public MapColor? _envColorWhite;
public MapColor? _envColorLeftBoost;
public MapColor? _envColorRightBoost;
public MapColor? _envColorWhiteBoost;
public MapColor? _obstacleColor;
public int? _beatmapColorSchemeIdx;
public int? _environmentNameIdx;
//PinkCore Port
public bool? _oneSaber;
public bool? _showRotationNoteSpawnLines;
//Tags
public string[] _styleTags;
}
}
from memorypack.
Thank you for the reproduction code.
I have looked into it and it seems that your data has very long strings, over about 15KB.
(e.g. _difficultyLabel
.)
Since the default buffer size is 8KB, the buffer would overflow.
Check the argument for MemoryPackStreamingSerializer.DeserializeAsync.
The following seems to work:
MemoryPackStreamingSerializer.DeserializeAsync<ExtraSongData>(fileStream, bufferAtLeast: 65536, readMinimumSize: 65536))
bufferAtLeast
is the minimum buffer required. It must be longer than the maximum length of the utf8 encoding of your string. Try specifying a larger value.
from memorypack.
I did try to specify a larger buffer already... but I guess it wasn't big enough. Thanks for taking the time to look into it, I appreciate it.
from memorypack.
Related Issues (20)
- SerializationException: Sequence reached end, cannot provide more buffer HOT 2
- generator not work on other class
- Any hope for supporting netstandard2.0? HOT 1
- Serialize in Roslyn. HOT 5
- How do I use MemoryPack inside a source generator? HOT 2
- Support for signalR HOT 3
- [Unity 6 Beta] "~MemoryPack.Generator.dll' will not be loaded... HOT 1
- Can't Install With githubUrl in Unity HOT 6
- error cs0111, cs0579 HOT 2
- Unity Compile Error After Install MemoryPack.UnityShims
- How to define custom collection with shared property member HOT 1
- Is it possible to serialize a collection with IConvertible as keys?
- Can i get some help understanding the results of this benchmark ?
- Is it possible to utilise MemoryPack to compare serialized data of the same type? HOT 1
- [Question] Why are unions only supported for class and not struct, and is there a work around? HOT 1
- POCO structs are treated as unmanaged despite GenerateType.VersionTolerant HOT 5
- Platform implement HOT 1
- Support WebGL? HOT 1
- Run In Roslyn HOT 1
- Unions generate code that is not C#7 compliant HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from memorypack.