Comments (4)
@pavlexander Thanks for the investigation & the optimized code. You're right with the 1st optimization, LINQ is lazy-evaluated, the Where clause have been evaluated every Any(), First(), Last(), Sum(), Min(), Max() operation.
A traditional for-loop (your approach) to create a new list is also a solver to this problem. I think ToList() after the Where clause may also work like the traditional loop but i have to verify.
You're also right with the 2nd optimization, the Where clause in each while loop is too costly, using your approach (iterate candles through the list) is better.
The part will be updated maybe in this weekend. Again, thanks for your contribution :D
from trady.
@pavlexander The code is patched. Please see if the performance issue has fixed 😃
from trady.
@lppkarl yes, I can verify that now it's 437x faster than previous implementation :)
Thank you for taking care of this issue and of course, for Trady! You are doing a great job.
from trady.
Here, I have made an example, which will ONLY work for 1Minute to 2Hours transformation.
- Default time taken to execute: 150,000 ms (records: 3,221)
- Time taken to execute after optimizations Nr.1: 29,000 ms (records: 3,243)
- Time taken to execute after optimizations Nr.2: 132 ms (records: 3,242)
Old code:
public static IReadOnlyList<IOhlcv> Transform<TSourcePeriod, TTargetPeriod>(this IEnumerable<IOhlcv> candles)
where TSourcePeriod : IPeriod
where TTargetPeriod : IPeriod
{
if (typeof(TSourcePeriod).Equals(typeof(TTargetPeriod)))
return candles.ToList();
if (!IsTimeframesValid<TSourcePeriod>(candles, out IOhlcv err))
throw new InvalidTimeframeException(err.DateTime);
if (!IsTransformationValid<TSourcePeriod, TTargetPeriod>())
throw new InvalidTransformationException(typeof(TSourcePeriod), typeof(TTargetPeriod));
var outIOhlcvDatas = new List<IOhlcv>();
var periodInstance = Activator.CreateInstance<TTargetPeriod>();
var startTime = candles.First().DateTime;
while (startTime <= candles.Last().DateTime)
{
var nextStartTime = periodInstance.NextTimestamp(startTime);
if (periodInstance.IsTimestamp(startTime))
{
var outIOhlcvData = ComputeIOhlcvData(candles, startTime, nextStartTime);
if (outIOhlcvData != null)
outIOhlcvDatas.Add(outIOhlcvData);
}
startTime = nextStartTime;
}
return outIOhlcvDatas;
}
private static IOhlcv ComputeIOhlcvData(IEnumerable<IOhlcv> candles, DateTimeOffset startTime, DateTimeOffset endTime)
{
var candle = candles.Where(c => c.DateTime >= startTime && c.DateTime < endTime);
if (candle.Any())
{
var dateTime = candle.First().DateTime;
var open = candle.First().Open;
var high = candle.Max(stick => stick.High);
var low = candle.Min(stick => stick.Low);
var close = candle.Last().Close;
var volume = candle.Sum(stick => stick.Volume);
return new Candle(dateTime, open, high, low, close, volume);
}
return null;
}
New code (optimizations Nr. 2)
public static IReadOnlyList<IOhlcv> Transform<TSourcePeriod, TTargetPeriod>(this IEnumerable<IOhlcv> candles)
where TSourcePeriod : IPeriod
where TTargetPeriod : IPeriod
{
if (typeof(TSourcePeriod).Equals(typeof(TTargetPeriod)))
return candles.ToList();
if (!IsTimeframesValid<TSourcePeriod>(candles, out IOhlcv err))
throw new InvalidTimeframeException(err.DateTime);
if (!IsTransformationValid<TSourcePeriod, TTargetPeriod>())
throw new InvalidTransformationException(typeof(TSourcePeriod), typeof(TTargetPeriod));
var outIOhlcvDatas = new List<IOhlcv>();
var startTime = candles.First().DateTime;
var periodStart = GetPeriodStart(startTime);
var periodEnd = periodStart.AddHours(2);
List<IOhlcv> inputCandles = candles.ToList();
List<IOhlcv> candle = new List<IOhlcv>();
for (int i = 0; i < inputCandles.Count(); i++)
{
var dt = inputCandles[i].DateTime;
if (dt >= periodEnd)
{
periodStart = periodEnd;
periodEnd = periodStart.AddHours(2);
ComputeIOhlcvDataPreprocessed(outIOhlcvDatas, candle);
candle = new List<IOhlcv>();
}
candle.Add(inputCandles[i]);
}
if (candle.Any())
{
ComputeIOhlcvDataPreprocessed(outIOhlcvDatas, candle);
candle = null;
}
return outIOhlcvDatas;
}
private static DateTimeOffset GetPeriodStart(DateTimeOffset dt)
{
return new DateTimeOffset(dt.Year, dt.Month, dt.Day, dt.Hour, 0, 0, dt.Offset);
}
private static void ComputeIOhlcvDataPreprocessed(List<IOhlcv> result, IEnumerable<IOhlcv> candle)
{
if (candle.Any())
{
var dateTime = candle.First().DateTime;
var open = candle.First().Open;
var high = candle.Max(stick => stick.High);
var low = candle.Min(stick => stick.Low);
var close = candle.Last().Close;
var volume = candle.Sum(stick => stick.Volume);
result.Add(new Candle(dateTime, open, high, low, close, volume));
}
}
And again, I will repeat myself - the code is done for demo purposes. It has some hard-coded periods. You are free to modify it to make a universal solution out of that.
It's just to show that there is a huge room for improvement.
from trady.
Related Issues (20)
- Transforming Data issue HOT 1
- Not getting expected CCI values HOT 3
- update problem HOT 3
- Kst indicator
- commodity channe lindex incorrect value
- Minute to hour transformation: failure HOT 3
- Rule bug?
- Performance when computing single indexes on RSI with large datasets HOT 2
- How to correctly call and compute indicators HOT 1
- Library incompatible w/ newer versions of CsvHelper HOT 6
- Volume Profile Indicator HOT 1
- Takes Too Long To Get Data From Importer HOT 1
- Need Help With Custom Importer HOT 11
- Indicator on Indicator 'TInput TOutput not matched' HOT 1
- TransformToCandles<Daily> Return repeated days
- Trady for crypto - Node.js
- AccumulationDistributionLine<TInput,TOutput> constructor is protected and cannot be instanciated!
- Performance issues with 2000 candles & Backtesting
- Error occured using trade
- How to import trades
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trady.