Code Monkey home page Code Monkey logo

geta-optimizely-productfeed's People

Contributors

jevgenijsp avatar marisks avatar mattisolsson avatar valdisiljuconoks avatar yungis avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

geta-optimizely-productfeed's Issues

Proposal: Refactor to support processing pipelines, different exporters and servants

Background

It was required to have like 3-4 different product feeds. The current package generates feed on the fly and affects the site performance quite a lot. If it had been done at least 4 times, the site would be down. Also, it does not cache things and the same feed can be requested / generated multiple times.

Proposal

Implement differently:

  1. loads Commerce content once (collector)
  2. enriches it with "expensive" data (like, stock info) once (processing pipeline)
  3. runs collected data through 4 different "converters" that each output a file (CSV, XML, JSON, whatever) and saves in BLOB storage (exporters)
  4. there is a separate API that serves the latest feed file from the BLOB (servant)

Add progress logging

The feed generation process often takes quite a long time and it is frustrating to wait for it to finish without any indication about its progress.

It would be great to have some progress indicator that updates the job status bar every now and then.

Should be quite an easy change since we have loaded things to process in a list already.

SQL Execution timeout expired

Got SQL Execution timeout when FeedEntity has ~40MB size.
This error is discoverable only in hosted environments and with 100% reproducibility.

Log:

Error occurred. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
ClientConnectionId:5967d717-c8d8-4be5-b5ff-3c1c21a1b6e9
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:430c2ed5-d826-4e1b-b83b-4d862311cbe9
Routing Destination:d6914a419bd2.tr371.norwayeast1-a.worker.database.windows.net,11049
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable`1 commandBatches, IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList`1 entriesToSave)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(StateManager stateManager, Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.<>c.b__104_0(DbContext _, ValueTuple`2 t)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Geta.Optimizely.ProductFeed.Repositories.FeedRepository.Save(ICollection`1 feedData)
at Geta.Optimizely.ProductFeed.ProcessingPipeline`1.Process(JobStatusLogger logger, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Geta.Optimizely.ProductFeed.FeedBuilderCreateJob.Execute()

Any suggestions to fix this?
As a temp solution, I think we can override the FeedDbContext and increase the command timeout.

public static class FeedApplicationDbContextFactory
{
    public static FeedApplicationDbContext CreateDbContext(string connectionString)
    {
        var optionsBuilder = new DbContextOptionsBuilder<FeedApplicationDbContext>();
        optionsBuilder.UseSqlServer(connectionString, b =>
        {
            b.CommandTimeout(int.MaxValue);
        });

        return new FeedApplicationDbContext(optionsBuilder.Options);
    }
}

And replace the FeedApplicationDbContext service like this after the services.AddProductFeed() registration:
services.Replace(ServiceDescriptor.Transient(_ => FeedApplicationDbContextFactory.CreateDbContext(cmsOptions.ConnectionString)));

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.