Use GridFS about toolkit HOT 8 CLOSED

@zy4 Many jobs are stuck in the 'Generating output' state because of this issue. For now, I'll display a warning in the process log, when the total size of all JSONs belonging to a job is close to 16MB.

from toolkit.

Going to work on it. But there would be an even simpler solution. Why not storing json files in the job directory?

from toolkit.

storing the json in the database does only make sense if you want to get only parts out of it. but as far as I see, the json comes always in one piece.

from toolkit.

I have also been thinking the same. I guess even pagination is not going to be affected. We never seem to read the JSONs in parts. If we would only store them locally, GridFS is not even required

from toolkit.

Let's stick with GridFS, it gives us more options for future lazy loading features and indexing documents.

from toolkit.

I agree. Gives us more flexibility.

from toolkit.

Many jobs in the course crashed because the output JSONs were too big to be written into the database; we should perhaps switch to not storing them in the DB.

from toolkit.

I recommend just to store them as json files in the jobfolder. Having them in GridFS is of no advantage because the file content is not indexed. The only advantage would be to allow for easier database migrations but then we should store the whole jobfolder in GridFS which is bad for debugging. And also we never migrate the database. So I will just close this issue and reopen a new one.

from toolkit.