Overview
Ever since upgrading to 0.4.0 with Hangfire at 1.8.7, we have been seeing more errors from the system and some intermittent AccessViolation exceptions on Windows systems stemming from Hangfire. To be clear, on Linux, we don't see the AccessViolation exceptions but we do see new errors that weren't there before. Using Hangfire 1.8.7 with SQLite 0.3.4 seems to have no obvious issues.
Some additional information
I am not specifying any special properties on the SQLiteStorageOptions
during setup, so all settings are based on the default in both 0.3.4 and 0.4.0 i.e. using WAL and pool size of 20.
Thinking about the problem
From my perspective, I think it's possible that this could be related to sqlite-net 1003. In 0.4.0, there was a major change to use a pooled DB connection strategy. I think it's possible that, if someone were to not dispose after fetching the IStorageConnection
, they could unintentionally lead to a situation where the IStorageConnection
(i.e. HangfireSQLiteConnection
) is actually garbage collected instead of disposed. That may trigger similar behavior as in the issue seen in the library.
Since this is a library to be consumed by Hangfire itself, it may not always be possible to guarantee correctness on "their side of the fence." From my brief review, I couldn't find anything else that really struck me as a possibility for having the finalizer be called without disposing beforehand.
Thoughts on solutions
From my decidedly narrow perspective, I think there are two easy paths:
- Do not wait to attain a reference to a created DbConnection. If we grab a reference to it immediately, you can prevent the GC collecting it without a Dispose being called if that reference never came back into the queue via
EnqueueOrPhaseOut
, this would prove beyond a doubt that something did not call Dispose when it should have on the IStorageConnection
.
- Do not pass the DbConnection straight into the constructor of
HangfireSQLiteConnection
. Instead, pass either a function/lambda to fetch a DbConnection in the implementations and have a proper using
in place each time. This eliminates the need for worrying about other people correctly disposing of things leading to such serious situations.
Things I'm not as confident about:
- Is it possible that these issues are related more to using WAL vs. journal? I think this is unlikely given how well tested SQLite is, but I am not an expert.
- Would avoiding
sqlite-net-pcl
and using Microsoft.Data.Sqlite
be an alternate path? Maybe it doesn't solve correctness issues, but maybe it avoids having AccessViolation exceptions during collection.
- Am I totally off base in thinking it's related to the issue in
sqlite-net-pcl
when there's an alternate explanation?
I am happy to produce a PR for one of paths, but I'd like to hear which path would be desired.
Errors
Log messages on Linux that weren't there in 0.3.4
2024-01-14 01:24:39.914 +00:00 [ERR] Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09
SQLite.SQLiteException: database is locked
at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)
at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)
at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)
at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
at Hangfire.Server.RecurringJobScheduler.EnqueueNextRecurringJobs(BackgroundProcessContext context)
at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)
at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)
2024-01-14 01:24:43.899 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790378:state-lock'. The resource is not locked or is locked by another owner.
2024-01-14 01:24:46.916 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790386:state-lock'. SQLite.SQLiteException: database is locked
2024-01-15 00:01:33.303 +00:00 [ERR] 10 state change attempt(s) failed due to an exception, moving job to the FailedState
SQLite.SQLiteException: database disk image is malformed
at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context)
at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContext context, IStorageConnection connection, String jobId, IState state, IReadOnlyDictionary`2 customData, String[] expectedStates, IFetchedJob completeJob,
Exceptions running on Windows with .NET 8.0.1:
All of these stacks ended with an AccessViolation leading to a total application failure.
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLite.PreparedSqlLiteInsertCommand.Finalize()
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLite.SQLite3.Finalize(SQLitePCL.sqlite3_stmt)
at SQLite.PreparedSqlLiteInsertCommand.Dispose(Boolean)
at SQLite.PreparedSqlLiteInsertCommand.Finalize()
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
at SQLitePCL.SQLite3Provider_e_sqlite3.SQLitePCL.ISQLite3Provider.sqlite3_open_v2(SQLitePCL.utf8z, IntPtr ByRef, Int32, SQLitePCL.utf8z)
at SQLitePCL.raw.sqlite3_open_v2(SQLitePCL.utf8z, SQLitePCL.sqlite3 ByRef, Int32, SQLitePCL.utf8z)
at SQLitePCL.raw.sqlite3_open_v2(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
at SQLite.SQLite3.Open(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
at SQLite.SQLiteConnection..ctor(SQLite.SQLiteConnectionString)
at SQLite.SQLiteConnection..ctor(System.String, SQLite.SQLiteOpenFlags, Boolean)
at Hangfire.Storage.SQLite.SQLiteStorage+<>c__DisplayClass8_0.<.ctor>b__0()
at Hangfire.Storage.SQLite.SQLiteStorage.CreateAndOpenConnection()
at Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext)
at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object)
at Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object)
at Hangfire.Processing.BackgroundDispatcher.DispatchLoop()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)