agjacome / mahmi Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 25.16 MB

Mechanism of action of the human microbiome

Home Page: http://sing.ei.uvigo.es/

License: Other

Java 100.00%

mahmi's People

Contributors

Watchers

mahmi's Issues

Fasta with large sequences causes exception on database write

Steps to reproduce:

Load project containing metagenome fastas with large sequences. Example: "Twin Gut Microflora Study".
Wait for the error to pop out.

This happens just after fa5bf5f. Previous versions were valid.

Relevant log output:

ERROR [ForkJoinPool.commonPool-worker-1] ProjectService.java:131 - Error while loading project
java.util.concurrent.CompletionException: es.uvigo.ei.sing.mahmi.loader.LoaderException: es.uvigo.ei.sing.mahmi.database.daos.DAOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help. In current row format, BLOB prefix of 768 bytes is stored inline.
    at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:205) ~[na:1.8.0_31]
    at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:459) [na:1.8.0_31]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [na:1.8.0_31]
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:902) [na:1.8.0_31]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1689) [na:1.8.0_31]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1644) [na:1.8.0_31]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [na:1.8.0_31]
Caused by: es.uvigo.ei.sing.mahmi.loader.LoaderException: es.uvigo.ei.sing.mahmi.database.daos.DAOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help. In current row format, BLOB prefix of 768 bytes is stored inline.
    at es.uvigo.ei.sing.mahmi.loader.LoaderException.withCause(LoaderException.java:24) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.daoAction(ProjectLoaderController.java:123) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.insertMetaGenome(ProjectLoaderController.java:101) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.loadFastas(ProjectLoaderController.java:59) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.lambda$1(ProjectLoaderController.java:70) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController$$Lambda$63/539522838.accept(Unknown Source) ~[na:na]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_31]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.loadProjectFiles(ProjectLoaderController.java:70) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.lambda$0(ProjectLoaderController.java:49) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController$$Lambda$47/1866691035.run(Unknown Source) ~[na:na]
    at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:454) [na:1.8.0_31]
    ... 5 common frames omitted
Caused by: es.uvigo.ei.sing.mahmi.database.daos.DAOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help. In current row format, BLOB prefix of 768 bytes is stored inline.
    at es.uvigo.ei.sing.mahmi.database.daos.DAOException.withCause(DAOException.java:24) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO.action(MySQLAbstractDAO.java:129) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO.write(MySQLAbstractDAO.java:89) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO.insert(MySQLAbstractDAO.java:62) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLMetaGenomesDAO.insert(MySQLMetaGenomesDAO.java:1) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.lambda$4(ProjectLoaderController.java:101) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController$$Lambda$64/889608535.get(Unknown Source) ~[na:na]
    at es.uvigo.ei.sing.mahmi.loader.ProjectLoaderController.daoAction(ProjectLoaderController.java:121) ~[classes/:na]
    ... 14 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help. In current row format, BLOB prefix of 768 bytes is stored inline.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_31]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_31]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_31]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408) ~[na:1.8.0_31]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.Util.getInstance(Util.java:360) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2077) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2062) ~[mysql-connector-java-5.1.34.jar:5.1.34]
    at com.zaxxer.hikari.proxy.PreparedStatementProxy.executeUpdate(PreparedStatementProxy.java:61) ~[HikariCP-2.3.1.jar:na]
    at com.zaxxer.hikari.proxy.PreparedStatementJavassistProxy.executeUpdate(PreparedStatementJavassistProxy.java) ~[HikariCP-2.3.1.jar:na]
    at es.uvigo.ei.sing.mahmi.database.utils.FunctionalJDBC.lambda$19(FunctionalJDBC.java:202) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.utils.FunctionalJDBC$$Lambda$45/529310321.apply(Unknown Source) ~[na:na]
    at es.uvigo.ei.sing.mahmi.database.utils.FunctionalJDBC$1.run(FunctionalJDBC.java:269) ~[classes/:na]
    at fj.control.db.DB$6.run(DB.java:106) ~[functionaljava-4.2.jar:na]
    at fj.control.db.DB$6.run(DB.java:106) ~[functionaljava-4.2.jar:na]
    at fj.control.db.DB$3.run(DB.java:64) ~[functionaljava-4.2.jar:na]
    at fj.control.db.DbState.run(DbState.java:100) ~[functionaljava-4.2.jar:na]
    at es.uvigo.ei.sing.mahmi.database.utils.FunctionalJDBC.withState(FunctionalJDBC.java:249) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.utils.FunctionalJDBC.runReadWrite(FunctionalJDBC.java:50) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO.lambda$1(MySQLAbstractDAO.java:89) ~[classes/:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO$$Lambda$39/13785621.f(Unknown Source) ~[na:na]
    at es.uvigo.ei.sing.mahmi.database.daos.mysql.MySQLAbstractDAO.action(MySQLAbstractDAO.java:125) ~[classes/:na]

System information:

$ uname -srvmo
Linux 3.18.6-1-ARCH #1 SMP PREEMPT Sat Feb 7 08:44:05 CET 2015 x86_64 GNU/Linux

$ java -v
openjdk version "1.8.0_31"
OpenJDK Runtime Environment (build 1.8.0_31-b13)
OpenJDK 64-Bit Server VM (build 25.31-b07, mixed mode)

$ mysql --version
mysql  Ver 15.1 Distrib 10.0.16-MariaDB, for Linux (x86_64) using readline 5.1

Refactor HttpApplication.getSingletons( )

Current HttpApplication.getSingletons() code is a bit of a mess. It will probably be nicer if all that DAOs and Services construction were replaced by a DAOFactory, LoaderFactory, CutterFactory and a ServiceFactory. Or something else that allows us to clean up that code in a simple and effective way.

Lombok-Maven integration fails at command-line compilation

Lombok integration with Eclipse works as expected, but direct compilation from command line ($ mvn compile) fails. Sometimes it is a val whose type cannot be correctly inferred, sometimes an annotation does not get processed, and sometimes even a NullPointerException or StackOverflowError is thrown during compilation. These problems are always related to lombok features and not programmer-induced errors, and using the "manual" version (e.g.: creating a getter method instead of using the @Getter annotation) solves them.

It is said that adding the lombok dependency to maven's pom.xml should suffice to make it work seamlessly. If something else is required, fix it. Otherwise, it is a Lombok bug, check if it has already been reported upstream, or report otherwise.

Preserve identifiers when reading a Fasta

Currently, FastaReader only reads sequences from Fasta files, discarding any identifier (lines preceded by >). This should not be the case, that information could be necessary in a future and thus it must be preserved.

This implies refactoring of Fasta, FastaReader, probably FastaWriter to write that info alongside the hash of each sequence, and probably our database in order to persist also those identifiers.

Allow automatic retrieval of metagenome projects from MGRast

We have quite a "manual" method at the moment to process metagenome projects: download from MGRast, store all files inside a server directory, and request the MAHMI system to process that directory.

This should be changed to automate the download of each project's files from MGRast. In a first stage, just providing the MGRast project ID is enough, but that should also be improved to automatic download of all projects identifiable by some given keywords (thus, searching in MGRast and downloading each matchin project).

Implement task management

Every long-running operation should be managed, allowing to be restarted from point of failure, know its state at any moment or cancel/pause/launch from last saved progress.

Implement automated tests for all testable code

Only AminoAcid and Nucleobase enums have automated tests at the moment. This should be changed, and every testable unit of code should have tests guaranteeing its correct behavior.

Already included dependencies:

JUnit v4.12 as basic testing framework
JUnit-QuickCheck v0.4 as property-based testing framework
Mockito v.1.10.19 as mocking framework

Basic guidelines:

Don't try to reach 100% coverage, it's unnecessary. Just test what requires to be tested, do not create a test for a getter that just returns an instace field.
Whenever a law or a universally quantified property can be stated, use junit-quickcheck capabilities to formulate it, and let it be auto-proven.
Whenever a test can be parameterizable, do parameterize it and not test with only one possible input/output.
Create fully independent tests. A unit test should never depend on other test nor in any global state or non-local assumption.
Use test doubles (mocks, stubs, etc) whenever the test outcome can be compromised by a failure in an external dependency. If "the real thing" can be used, prefer it always over a mock, but do not hesitate to use one when seems necessary or reasonable.

Add documentation to every non-private component

JavaDoc formatted comments should be added to every component (class, method, field, etc.). Private ones are not actually required.

Should MetaGenomeProteins be renamed to Translations?

We have Digestions instead of ProteinEnzymePeptides, shouldn't we have a name like Translations instead of MetaGenomeProteins?

Return semantically correct error codes in HTTP API

Current DatabaseEntityAbstractService.respond(...) method always returns a 500 Internal Server Error whenever a database error occurs. This should be replaced with something more meaningful, as in some cases it could be a client error (Bad Request) or something not yet contemplated.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.