Code Monkey home page Code Monkey logo

Comments (4)

karlwancl avatar karlwancl commented on July 16, 2024 1

@pavlexander Thanks for the investigation & the optimized code. You're right with the 1st optimization, LINQ is lazy-evaluated, the Where clause have been evaluated every Any(), First(), Last(), Sum(), Min(), Max() operation.

A traditional for-loop (your approach) to create a new list is also a solver to this problem. I think ToList() after the Where clause may also work like the traditional loop but i have to verify.

You're also right with the 2nd optimization, the Where clause in each while loop is too costly, using your approach (iterate candles through the list) is better.

The part will be updated maybe in this weekend. Again, thanks for your contribution :D

from trady.

karlwancl avatar karlwancl commented on July 16, 2024 1

@pavlexander The code is patched. Please see if the performance issue has fixed 😃

from trady.

pavlexander avatar pavlexander commented on July 16, 2024 1

@lppkarl yes, I can verify that now it's 437x faster than previous implementation :)

Thank you for taking care of this issue and of course, for Trady! You are doing a great job.

from trady.

pavlexander avatar pavlexander commented on July 16, 2024

Here, I have made an example, which will ONLY work for 1Minute to 2Hours transformation.

  • Default time taken to execute: 150,000 ms (records: 3,221)
  • Time taken to execute after optimizations Nr.1: 29,000 ms (records: 3,243)
  • Time taken to execute after optimizations Nr.2: 132 ms (records: 3,242)

Old code:

public static IReadOnlyList<IOhlcv> Transform<TSourcePeriod, TTargetPeriod>(this IEnumerable<IOhlcv> candles)
	where TSourcePeriod : IPeriod
	where TTargetPeriod : IPeriod
{
	if (typeof(TSourcePeriod).Equals(typeof(TTargetPeriod)))
		return candles.ToList();

	if (!IsTimeframesValid<TSourcePeriod>(candles, out IOhlcv err))
		throw new InvalidTimeframeException(err.DateTime);

	if (!IsTransformationValid<TSourcePeriod, TTargetPeriod>())
		throw new InvalidTransformationException(typeof(TSourcePeriod), typeof(TTargetPeriod));

	var outIOhlcvDatas = new List<IOhlcv>();
	var periodInstance = Activator.CreateInstance<TTargetPeriod>();
	
	var startTime = candles.First().DateTime;
	while (startTime <= candles.Last().DateTime)
	{
		var nextStartTime = periodInstance.NextTimestamp(startTime);
		if (periodInstance.IsTimestamp(startTime))
		{
			var outIOhlcvData = ComputeIOhlcvData(candles, startTime, nextStartTime);
			if (outIOhlcvData != null)
				outIOhlcvDatas.Add(outIOhlcvData);
		}
		startTime = nextStartTime;
	}

	return outIOhlcvDatas;
}

private static IOhlcv ComputeIOhlcvData(IEnumerable<IOhlcv> candles, DateTimeOffset startTime, DateTimeOffset endTime)
{
	var candle = candles.Where(c => c.DateTime >= startTime && c.DateTime < endTime);

	if (candle.Any())
	{
		var dateTime = candle.First().DateTime;
		var open = candle.First().Open;
		var high = candle.Max(stick => stick.High);
		var low = candle.Min(stick => stick.Low);
		var close = candle.Last().Close;
		var volume = candle.Sum(stick => stick.Volume);
		return new Candle(dateTime, open, high, low, close, volume);
	}
	return null;
}

New code (optimizations Nr. 2)

public static IReadOnlyList<IOhlcv> Transform<TSourcePeriod, TTargetPeriod>(this IEnumerable<IOhlcv> candles)
	where TSourcePeriod : IPeriod
	where TTargetPeriod : IPeriod
{
	if (typeof(TSourcePeriod).Equals(typeof(TTargetPeriod)))
		return candles.ToList();

	if (!IsTimeframesValid<TSourcePeriod>(candles, out IOhlcv err))
		throw new InvalidTimeframeException(err.DateTime);

	if (!IsTransformationValid<TSourcePeriod, TTargetPeriod>())
		throw new InvalidTransformationException(typeof(TSourcePeriod), typeof(TTargetPeriod));

	var outIOhlcvDatas = new List<IOhlcv>();
	
	var startTime = candles.First().DateTime;
	var periodStart = GetPeriodStart(startTime);
	var periodEnd = periodStart.AddHours(2);
	List<IOhlcv> inputCandles = candles.ToList();
	List<IOhlcv> candle = new List<IOhlcv>();
	for (int i = 0; i < inputCandles.Count(); i++)
	{
		var dt = inputCandles[i].DateTime;
		if (dt >= periodEnd)
		{
			periodStart = periodEnd;
			periodEnd = periodStart.AddHours(2);

			ComputeIOhlcvDataPreprocessed(outIOhlcvDatas, candle);
			candle = new List<IOhlcv>();
		}

		candle.Add(inputCandles[i]);
	}

	if (candle.Any())
	{
		ComputeIOhlcvDataPreprocessed(outIOhlcvDatas, candle);
		candle = null;
	}

	return outIOhlcvDatas;
}

private static DateTimeOffset GetPeriodStart(DateTimeOffset dt)
{
	return new DateTimeOffset(dt.Year, dt.Month, dt.Day, dt.Hour, 0, 0, dt.Offset);
}

private static void ComputeIOhlcvDataPreprocessed(List<IOhlcv> result, IEnumerable<IOhlcv> candle)
{
	if (candle.Any())
	{
		var dateTime = candle.First().DateTime;
		var open = candle.First().Open;
		var high = candle.Max(stick => stick.High);
		var low = candle.Min(stick => stick.Low);
		var close = candle.Last().Close;
		var volume = candle.Sum(stick => stick.Volume);

		result.Add(new Candle(dateTime, open, high, low, close, volume));
	}
}

And again, I will repeat myself - the code is done for demo purposes. It has some hard-coded periods. You are free to modify it to make a universal solution out of that.

It's just to show that there is a huge room for improvement.

from trady.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.