Comments (28)
theotherp
commented on May 24, 2024
Hmm. Please post the debug infos.
Do you run the migration by calling Hydra via URL? If yes, could you please shutdown v1 and use the file based migration and see if that works?
from nzbhydra2.
fryfrog
commented on May 24, 2024
This is via URL, I've tried via file too and it is similar but I hadn't stopped nzbhydra when I did it. I'll try that now.
from nzbhydra2.
fryfrog
commented on May 24, 2024
nzbhydra-debuginfos-2018-01-06-14-42.zip
from nzbhydra2.
fryfrog
commented on May 24, 2024
Pointed at files with nzbhydra stopped, doesn't look much different. Maybe I'm missing some required dependency?
java.sql.SQLException: The database has been closed
at org.sqlite.core.NativeDB.throwex(NativeDB.java:471)
at org.sqlite.core.NativeDB.errmsg_utf8(Native Method)
at org.sqlite.core.NativeDB.errmsg(NativeDB.java:137)
at org.sqlite.core.DB.newSQLException(DB.java:921)
at org.sqlite.core.DB.throwex(DB.java:886)
at org.sqlite.core.NativeDB._open_utf8(Native Method)
at org.sqlite.core.NativeDB._open(NativeDB.java:71)
at org.sqlite.core.DB.open(DB.java:174)
at org.sqlite.core.CoreConnection.open(CoreConnection.java:220)
at org.sqlite.core.CoreConnection.(CoreConnection.java:76)
at org.sqlite.jdbc3.JDBC3Connection.(JDBC3Connection.java:26)
at org.sqlite.jdbc4.JDBC4Connection.(JDBC4Connection.java:24)
at org.sqlite.SQLiteConnection.(SQLiteConnection.java:45)
at org.sqlite.JDBC.createConnection(JDBC.java:114)
at org.sqlite.JDBC.connect(JDBC.java:88)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at org.nzbhydra.migration.SqliteMigration.migrate(SqliteMigration.java:92)
at org.nzbhydra.migration.SqliteMigration$$FastClassBySpringCGLIB$$72e2874a.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.nzbhydra.migration.SqliteMigration$$EnhancerBySpringCGLIB$$7dae4580.migrate()
at org.nzbhydra.migration.FromPythonMigration.startMigration(FromPythonMigration.java:105)
at org.nzbhydra.migration.FromPythonMigration.migrateFromFiles(FromPythonMigration.java:66)
at org.nzbhydra.migration.FromPythonMigration$$FastClassBySpringCGLIB$$25126370.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.nzbhydra.migration.FromPythonMigration$$EnhancerBySpringCGLIB$$7c1da1b2.migrateFromFiles()
at org.nzbhydra.migration.MigrationWeb.migrateFromFiles(MigrationWeb.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.nzbhydra.auth.HydraAnonymousAuthenticationFilter.doFilter(HydraAnonymousAuthenticationFilter.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Close
from nzbhydra2.
theotherp
commented on May 24, 2024
Hm no. How big is your nzbhydra.db database file (in MB)?
from nzbhydra2.
fryfrog
commented on May 24, 2024
-rw------- 1 nzbhydra nzbhydra 1.2G Jan 6 13:51 nzbhydra.db
Holy shit, can I trim that down w/ nzbhydra? :/
from nzbhydra2.
theotherp
commented on May 24, 2024
Jesus... Mine is 200M and I thought that was large...
from nzbhydra2.
theotherp
commented on May 24, 2024
If you have enough RAM can try XMX 4096 or even higher. No idea how long the migration would take.
Trimming down would depend on what you want to keep. I could give you a script to remove everything that's older than X days. You'd lose the stats from those entries. Your call.
from nzbhydra2.
fryfrog
commented on May 24, 2024
I'd be fine w/ removing everything older than X days. I'll also try bumping up the memory. The migration fails instantly as is, though.
from nzbhydra2.
theotherp
commented on May 24, 2024
Yeah, I think it just has a heart attack at the sight of that db file. I have some other reports and my head is nearly falling on my keyboard, so the script will have to wait until tomorrow ;-)
from nzbhydra2.
fryfrog
commented on May 24, 2024
Sounds good :)
from nzbhydra2.
fryfrog
commented on May 24, 2024
I ran all the way up to 16G and it still fails, leaving until whenever too :)
from nzbhydra2.
theotherp
commented on May 24, 2024
I hope it actually is the size... Need to take care to write a script that can handle that size in the first place... But apparently v1 can do it just fine for some reason.
from nzbhydra2.
fryfrog
commented on May 24, 2024
Maybe the last "feature" for nzbhydra could be a trim/clean db? :)
from nzbhydra2.
fryfrog
commented on May 24, 2024
Maybe a settings only import would be a feasible alternative? I'm mainly lazy and just don't want to re-enter all my indexers. :)
from nzbhydra2.
theotherp
commented on May 24, 2024
It's in the works.
Am 07.01.2018 16:30 schrieb "Donald Webster" <[email protected]>:
… Maybe a settings only import would be a feasible alternative? I'm mainly
lazy and just don't want to re-enter all my indexers. :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ANtAeQsv-xXu_kLtbqjXd8ClcRgXTyOCks5tIOMQgaJpZM4RVdnF>
.
from nzbhydra2.
wizzy99
commented on May 24, 2024
I was unsuccessful in migrating a 4.2GB database (2493325 searches) over. Ran for an hour via url and then ran out of memory. I then upped the memory from 128->512 and tried file based, but am unsure if it was before or after I upgraded to the version that supports indexer only migration, and got a garbage collection error. Retried with the url on the new version and got the same GC error. Finally settled on just migrating the indexers.
from nzbhydra2.
theotherp
commented on May 24, 2024
4.2GB is far beyond anything I've expected. Jesus, what do you people do... My database is 200MB and I thought that was large.
I'll have to test how v2 behaves with databases so large. That's enormous. I might have to either rethink my database model or support "proper" databases that are run externally.
from nzbhydra2.
fryfrog
commented on May 24, 2024
I'm blind, where can I find the indexer/settings only migration? Via file?
from nzbhydra2.
theotherp
commented on May 24, 2024
Checkbox at the bottom. If you don't see that checkbox try Shift-F5. If that helps please tell me because that would mean there's a problem with the caching :-(
from nzbhydra2.
fryfrog
commented on May 24, 2024
Thanks, I was indeed blind! :)
from nzbhydra2.
theotherp
commented on May 24, 2024
I'm glad ;-)
from nzbhydra2.
fryfrog
commented on May 24, 2024
It worked and I'm okay w/ closing this thread. Alternatively, I can pass you my giant .db for processing if needed.
from nzbhydra2.
theotherp
commented on May 24, 2024
Well, that would certainly help a lot! Please send me a link via email to [email protected]
from nzbhydra2.
fryfrog
commented on May 24, 2024
Email w/ link sent. :)
from nzbhydra2.
theotherp
commented on May 24, 2024
I've just successfully migrated your database using XMX=4096
Not sure why it doesn't work for you. I'll do some tests how much memory is needed to actually use Hydra with that database. Not sure how it will run. When I'm finished and everything works I can send you the database if you want. But it may be that I will have to rethink my data model and throw away searches after some time. I don't think anyone needs a search history that spans two years...
from nzbhydra2.
fryfrog
commented on May 24, 2024
I wouldn't dedicate any time to this, I migrated just settings and that was plenty for me. Having a time based prune seems like a reasonable idea, especially making it configurable. I'll let you decide if you want to close this thread, but I would. :)
from nzbhydra2.
theotherp
commented on May 24, 2024
Alright.
The database validation took 10 minutes on the first startup after migration after that everything worked just fine with "only" 512MB memory. No major decrease in performance and even the stats generation was reasonably fast.
So thanks for the test data. It looks v2 can handle large databases good enough.
from nzbhydra2.
Related Issues (20)
- [Binsearch] Unable to find main table in binsearch page. This happens sometimes ;-) (HAPPENS ALWAYS)
- RSS feed doesn't return any results? HOT 6
- Is there a way to add Prowlarr indexers to nzbhydra2? HOT 2
- Unsecured Dev Endpoints HOT 2
- 5.3.9 Generic fails to start, stuck in startup loop after update from 5.3.8 (MAC OS) HOT 2
- Can't run NZBHydra Windows, systray icon appears then disappears, service starts then stops HOT 47
- Windows v5.3.9 won't start HOT 1
- 5.3.9 python libc on debian failed to start HOT 2
- Core.exe CPU Usage HOT 2
- can't log into Hydra2 anymore HOT 5
- NZBindex API version HOT 1
- nzbhydra leaving behind GB's in local temp folder HOT 1
- Feature Request: Add IP to History tables HOT 5
- MAC OS - wrappers report as outdated HOT 5
- can't send to Sabnzb after updating to 5.3.11 HOT 4
- Update through WebUI Doesn't Work HOT 3
- Incorrect issue number in changelog HOT 1
- Nzbhydra2 no longer launches on Ubuntu 22.04 after 6.0 update HOT 2
- REQ don't choose the smallest posting as the one fronting concatenated posts HOT 2
- Enhancement Request to Search History Table HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nzbhydra2.