esmero / archipelago-deployment Goto Github PK
View Code? Open in Web Editor NEWArchipelago Commons Docker Deployment Repository
Archipelago Commons Docker Deployment Repository
Additional options and configurations for Serials (newspapers, magazines, etc.) Digital Objects and Digital Object Collections
I found some broken links in the demo documentation and they are already fixed.
Following the installation directions for Installing Archipelago on Ubuntu (actually on a RHEL 7.6) for Beta3, seemed to have encountered the same issue as with Beta2. The difficulty appears when setting up Archipelago on a remote server (i.e. you are not using "http://localhost:8183/iiif/2/" as the IIIF Server base URL, but instead need to specify an actual base name.)
One should be able to specify the "Base URL of your IIIF Media Server public accesible [accessible] from the Outside World." from the IIIF Server Settings Form. This is not the case, since the form validation does not accept a valid URL of the IIIF server. A guess, is that the form validation for that field, might be querying for a IIIF media server at that URL, but it is querying on an INTERNAL docker network and not on the outward looking (non-virtual) network. The example given below, the URL provided does resolve in the web browser, but is not seen as valid in the form.
My current work around, is to modify the configuration files by following the steps located at: https://groups.google.com/d/msg/archipelago-commons/N5D0M6h7DUI/6xLYWfKUAwAJ
But that might be asking to much, for a first time Archipelago user.
Well, I am referencing docker-compose.dev.yml
here. It helps to launch the development php image and is used as an override file in docker-compose up
during development or debugging sessions. So I need to add it to the repo for testing and approval.
I am open to a different approach but for now, wish for you @DiegoPino to have access to the file.
Q: who has huge images? A: basically everyone!
Our cantaloupe custom Docker Container works well but there are some issues we may need to tune:
1.- When serving Files from S3, in specific large > 100 Mbytes TIFFS, the processor Streaming Strategy tends to fail randomly. It safer the Cache Strategy (means download and keep around) but also slower and requires more physical space on the server to keep the caches around. Also for these Java2D and Jair totally fail (like badly) and give me a strange Memory error. Using GraphicsMagic is safer but also slower and works. We can experiment and maybe do a test suit to know our limits/fine tuning needs?
2.- I created a new Docker Container Version that allows XMX and XMS to be passed as arguments and also include TurboJPEG and a newer version of GraphicsMagic.
For Natural Language Processing and AI analysis of extracted Corpus of text from Files, metadata Description fields or similar textual bodies i started building a few Search API Solr Post Processors that can deal with this, all under the umbrella of the Strawberryfield Runners module for 1.0.0. PS: Ping me for a showcase of the proof of concept i have running in AWS
But to make this an option for people testing-using or just admiring archipelago (yeah.. not sure how many) i want to dockerize these needs through an extra additional docker-compose.yml
that can be appended via the -f option when doing and docker-compose up -d
This
The idea here is basically not everyone needs fancy robots and AI getting names and links from their metadata, but some could need. I'm working on formalizing (and generalizing) the life code i have. I like the fact that it integrates good into the SBF runners idea but also forced me to make some changes there, like how/when things run and i found myself quite happy seeing that actually a lot can go directly to Solr instead of being saved on metadata but other things need to go into permanent storage.
Not the place (i know) but this also open a chance of working again on my idea that AI/NPL generated data needs to be tagged/classified and exposed to the world as such. Means i'm adding provenance, how it was processed, version of the software, etc to an upper structure of the JSON. That way we know and we can keep a separation of human generated classification v/s machine one. Machines are not always quite aware of context so the level of trust we can put on this data will vary a lot, so good we can pass that back to the end user.
@alliomeria because of metadata discussion and @giancarlobi since i think you will like this. Should be able to add code to https://github.com/esmero/strawberry_runners/tree/ISSUE-4 soon with the full refactor and the new plugins.I do have high hopes here!
This is really my own exploration. I'm thinking (while I think and code and do other stuff) on Fargate Tasks and fully automatic installations of Archipelago. I know what I'm doing and it feels Ok. Won't share details here until I have it running and solved. This does not apply to non Cloud based deployments and I do not think this is a good match for larger installations, but it does work for many smaller, medium use cases where every parts of the Stack (except SOLR which can be shared by many of even be a cloud service, Cantaloupe same idea via S3 Bucket separations) can be totally encapsulated, self sustained Containers that are totally regenerated in case of Software upgrades.
All this to serve low budget communities but also avoid spending my time ssh-ing every server in the world.
More later on this.
So. During my super production tests and while allowing a HUGE file (Gbyte+ to be delivered on demand to a viewer) to deal with a Range-IF
request I saw a few PHP Out of memory messages.
Guzzle was leaking memory through Symfony (binaryresponse).
NOTICE: PHP message: PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 989388968 bytes) in /var/www/html/vendor/guzzlehttp/psr7/src/Stream.php on line 225
Digging deeper and deeper happened to be a PHP bug (not our first one but also means we are stretching things quite a bit!)
See php/php-src#5014
The actual solution was merged in PHP 7.4.3+. So. Time to move forward if we want to support large large files. Related to this exploratory commit.
esmero/format_strawberryfield@f53adb4
What will happen?
> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:c88c3e29abfe9e657aa37c51ee123be73ca6d67ceb70512e25c21b0aba4f0a8b 0.0s
=> => naming to docker.io/esmero/php-7.4-fpm:1.0.0-RC2
But now some testing first!!
This was bothering Giancarlo for a few days. We are running 1.17 but the upgrade in his D9 instance killed any Field based Updates because of bad code in Search API.
More here https://www.drupal.org/project/search_api/issues/3178941#comment-13899506
I found an additional bug so had to make our own patch (c'mon Drupal..)
It would be great to have a system in Webforms for conditional logic like "Value is in" so that I can check and see if, for instance, the media type is a "3D Model", "Photograph", or "Video" instead of having 3 separate "Value is" for each media type.
See https://www.drupal.org/project/drupal/issues/1091852 and
See https://www.drupal.org/files/issues/2019-11-19/1091852-104.patch
Basically there is a Core bug in Drupal 8.x and 9.x that affects #state
driven forms when loaded/updated via AJAX. I remember (slightly so much code i have added in the last year) having to work double shifts to have our Ajax driven forms work top notch, but as we advance to more Complex webforms (the EAD 2002 one i'm doing is quite heavy on conditional logic) it could be cool to test if the patch here applies clean on 8.9.2 and if we really need it.
Yep. It took us some time, like almost 2 years from idea to what we have now and we designed Archipelago to be as compliant to Drupal 9 as we could with such a tiny team and too little time. Still, codebase has grown, we have so much good and complex too code, so we need to pass a general review on each Module we coded but also on each module we suggest should be used. This Issue will be a placeholder for this Friday, when that work will be passed. Idea is to check here what passed and open the issues needed to make the other ones pass. And then have (yes!) a 9.x-1.0 branch for our very own and of course more code to maintain... help!.
This question led to a fix from Cantaloupe (so thankful) and now we just need to update our Cantaloupe Docker container to 4.1.6 once that is out.
cantaloupe-project/cantaloupe#359
This will go into Beta3. Thanks
Related to #56
Giancarlo had the amazing idea of instead of depending on JSON-LD files that come and go, we provide for new deployments a smaller subset that can be used directly in
/admin/structure/strawberry_keynameprovider/schema_org/edit
We can also add a link in a README.md file inside strawberryfield module
This requires some group effort to select the best properties. Of course this is a starting point and people can extend or replace. But its a good and stable starting point
This base @context
file will be hosted in this repo and will have its own versioning.
Given the flexible nature archipelago people then can add other plugin instances via the UI to expand on further vocabularies
Thanks @giancarlobi
We did a few retests with our Deployment docs for Ubuntu and found some inconsistencies between our new Alpine Containers (in specific the FPM PHP 7.3 one, esmero-php) and Ubuntu's user ids.
After some cleaning up and spinning some machines up, we got it right again. Previous docs where fine for all-around-ubuntu, but we(i) wanted smaller containers right? π
Fix is up and docs/ubuntu.md was proof read, steps retraced and corrected. I also added some comments and explanations of why we do what we do there. Please give it a try.
By the way, On AWS on a M5.large EC2 machine, deployment took 5 minutes (including copy pasta and scrolling) and runs fast, blazing fast.
This is the the actual fix 7b811cf
Will keep this issue open until beta2 rolls.
Preparing for Beta3 (finally, gosh!)
@giancarlobi in case you want to keep in the meantime your archipelagos updated
docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/core:8.8.6'
docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/webform:^5.13;drush updatedb; drush cr'
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/moderated_content_bulk_publish:1.x-dev';drush en moderated_content_bulk_publish; drush updatedb;"
And for emails i'm using this
docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/smtp:^1.0'
I pretend for beta3 to deliver also a script that does this and enables via drush new modules to allow people to update without having to run a full composer install (if e.g they have custom modules)
Right now SBF Audio Formatter ships with the dimensions 720x240. This is fine and does not affect the functionality of the player, but it looks a bit funky when it loads for folks viewing an object.
...by making the height dimension smaller π€ΈββοΈ
I found that 720x50 pixels makes a nice and compact-looking player that also allows for maximum user-friendly scrubbing of the file.
Ok, this is new:
Since Jan 2020, if you have a quite old min.io and you are updating your archipelago to beta2 and your docker-compose by pulling latest containers, you could see (its a could, not an always) on your Archipelago (like no images and a lot of stress? You log into localhost:9000 and there is nothing there! You call your parents and nobody answers, panic!). Fear not! Probably your old container version did not upgrade cleanly.
This is what you would see if things are not well (other than missing assets which is what will alert you)
I'm running right now
minio/minio:RELEASE.2020-02-20T22-51-23Z
archipelago-deployment (8.x-1.0-beta2) $ docker logs esmero-minio
You are running an older version of MinIO released 1 week ago
Update: docker pull minio/minio:RELEASE.2020-02-20T22-51-23Z
Server startup failed with 'Unable to initialize config system: madmin: invalid AEAD algorithm ID'
Server switching to safe mode
Please use 'mc admin config' commands fix this issue
Endpoint: http://172.20.0.3:9000 http://172.21.0.3:9000 http://127.0.0.1:9000
Solution is simple, from your archipelago-deployment
folder do:
rm -rf miniodata/.minio.sys
Which will basically reset your configs to default.
Restart your docker ensemble
docker-compose restart
All good now.
We have to check if all Archipelago modules RC1 version are ok running onto Drupal 9.1
As an example deployment configuration of our basic JSON Key Name provider we were using https://Schema.org/Book.jsonld to filter a HUGE list of properties of the main Schema.org Context size to a manageable size. So yesterday i saw it go away. And it was hard!
Probably it has been gone for longer and since we do cache it i never noticed it. Schema.org is also inconsistent, why would you remove a public jsonld file???
So we need another subset of properties for the next deployment. Maybe even another strategy so filtering a larger context is simpler and more granular.
@giancarlobi. This can affect you too. even when not using docker.
Current copy is here https://github.com/esmero/archipelago-deployment/blob/8.x-1.0-beta2/web/sites/default/files/jsonld/c20db5b6a8d641e98c0570d1ef89017f.jsonld
This is our working enumeration of concrete tasks, per Component and Service, for public evaluation (and comments).
All tasks listed here are leading our full feature release end of 2019. Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.
There are a lot more things involved and many are already done and coded, but i wanted to have a list that was closer to features checklist than the actual code milestones we are managing internally. Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).
as:images
as:documents
, as:video
, as:sound
, as:dataset
elementsas:models
Media Displays
JSONAPI
on a set of Drush script to (Strawberry Seeds)
Just to share some settings here about this specific installation.
Today, Dec 4th 2019 Drupal 8.8.0 was released (yay! or nay?)
Full list of changes: https://www.drupal.org/project/drupal/releases/8.8.0
(read it, i did not.. and here i am)
Since 8.x-1.0-beta2 is coming i wanted to deliver by default that version. But again, so much has changed. Of course i tried the "do it, then solve it" way and i broken one of my local Archipelagos because of a Contributed Module (mentioned in the release notes, but who reads that right? jajaja). The Module is Drupal/pathalias
. Drupal 8.8.0 breaks on anything less than 1.6 and it needs to have 1.6 installed first before moving forward because of a Database/entity that needs to be deployed.
Site note: will have to use my Time machine tonight, i had some stuff i liked in that repo!
So until now: i managed to find this set of composer commands to move forward.
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/pathauto:^1.6'"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer remove webflo/drupal-core-require-dev"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require drupal/core-dev:^8.8"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require typo3/phar-stream-wrapper:3.1.3 pear/archive_tar:^1.4.8 drupal/core:8.8.0"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer update"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/pathauto:^1.6'"
docker exec -ti esmero-php bash -c "drush updatedb"
docker exec -ti esmero-php bash -c "drush cr"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer remove webflo/drupal-core-require-dev"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require drupal/core-dev:^8.8"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require typo3/phar-stream-wrapper:3.1.3 pear/archive_tar:^1.4.8 drupal/core:8.8.0"
docker exec -ti esmero-php bash -c "drush cr"
docker exec -ti esmero-php bash -c "drush en path_alias"
docker exec -ti esmero-php bash -c "drush updatedb"
docker exec -ti esmero-php bash -c "drush cr"
Updates: new core module that is needed (given by 8.8.0) drupal/path_alias.
Will document this, but gist is: needs to be either enabled on existing machines via drush, or on new ones our config/sync/core.extension.yml needs to have that one added. If not, BUM!
How lovely dependencies are....
As you also noticed i'm using php -dmemory_limit=-1 /usr/bin/composer
instead of composer
directly. Newer version of composer is using more than 512Mbytes we assigned to PHP. We will deal with that too.
So this are the first steps: i'm now testing a from scratch deployment. Once i'm ok with that i will move to an update and will provide also a new update script for the new release.
We also need to update config/sync settings and add new configurations. A lot do folks!
I'm super confused by this bug, but something in Drupal Search API after version 3.4 is breaking everything in our ensemble. Everything means, every time a Solr Index is called, either on a search result, an autocomplete, a config form or even a simple task like Seeing the http://localhost:8001/admin/config/search/search-api/index/default_solr_index
to find out who is tracked or not we get a nasty (white page) error like this:
NOTICE: PHP message: Uncaught PHP Exception LogicException: "The database connection is not serializable. This probably means you are serializing an object that has an indirect reference to the database connection. Adjust your code so that is not necessary. Alternatively, look at DependencySerializationTrait as a temporary solution." at /var/www/html/web/core/lib/Drupal/Core/Database/Connection.php line 1546
Which basically means, something (and i'm looking at you MR. SOLR SEARCH API π ) is injecting not an entityTypeManager
container, but probably an entityTypeManager->getStorage
as a property in a constructor, or, a specific logger
, or something that opens a connection to the DB
and keeps it in a property
initialized, as i already said, in the constructor.
Since many many things, like plugins, etc are serialized by the Symfony Kernel, we end with this quite huge error. Normally, if someone (not us) needs that, you either use a special trait that removes the container on __sleep() and restores it on __wake() (up), but i don't see the why this is now happening.
That said, it can be something we do/did, and since Solr Search API is now, maybe? trying to cache something or being smarter, it gets stuck in the way. I checked all our source code, like everything, and could not find a single place where we commit such Dependency Injection heresy (well one, in a config form, but that one is never called in a normal request stack and its this weeks code)
I spend 18 hours (with a small pause for Subway and lunch) and i can not find the culprit. π
Removed all our own dependencies, changed the index to only use Pages and Articles, removed our Formatters, our Events and of course all fields. And nothing! Rolled back to 3.4 and all is working, no errors, no messages, nothing, pure Archipelago beauty.
After this, i have to say it could be also any other contributed module interaction or the fact that we are trying to run Drupal core 8.8.1 too.
Fresh, well slept eyes. I would suggest, downloading 3.4 and 3.8 (latest), and doing and full release comparison to understand what changed. Only way we can deal with this since we will want eventually be able to move forward with the newer solarium library too.
Also, probably a from scratch deployment with 3.8 and only basic packages could give us a hint. Both options are quite time consuming (days, centuries) so i will move forward on beta2 with 3.4, but this task can not wait longer than beta3.
I have spoken. This is the way π
@marlo-longley @giancarlobi thoughts?
See esmero/archipelago-docker-images#8
New Cantaloupe container using Java 13 (OpenJDK) and Dec 29th 2019 Graphicsmagic + new settings.
We need to research if the new S3 settings i will add make things slower or not, i'm using for the first time chunked requests, could be fast on AWS S3, but slower on Mini.io. Time (or unhappy users) will tell.
Also moved some processors around, now PDF is dealt via PDFBOX, since Graphics magic uses the gs binary directly (or i got the compiling wrong..)
See also #5 and #35 for the whole historic recreation. @alliomeria this is for you so we can catch up with old business
This is our working enumeration of concrete tasks for this year, per Component and Service, for public evaluation (critics and comments welcome).
All tasks listed here are leading our full feature release somewhere this first half of the year. We went more ambitious (we planned for end of 2019) so code has been moving along and we feel we are quite there! Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.
Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).
as:images
as:documents
, as:video
, as:sound
, as:dataset
elementsas:models
Media Displays
JSONAPI
on a set of Drush script to (Strawberry Seeds)
Relating to esmero/archipelago_subtheme#14 in the subtheme, we pushed an updated schema.
This means some of the old settings were of the wrong type to match the new schema.
Update /config/sync/archipelago_subtheme.settings to match new schema types.
So we need to start not only thinking of making our docker-compose AWS smarter but actually our Docker Daemon π tuning smarter
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file
E.g I like to run my /root devices slim and fast. Means we need default volumen storage go into second drives.
And so on and so on.
NOTE: updated!
Let's add
See also #5 and #35 and #79 for the whole historic recreation. @alliomeria I will work on this tomorrow.
This is our working enumeration of concrete tasks until end of 2020 (the year of the void()), per Component and Service, for public evaluation (critics and comments welcome).
All tasks listed here are leading our full feature release somewhere this first half of the year. 2020 has been weird, hurtful but strangely productive and Archipelago sailing, floating steady in the current and full of suprises. Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.
Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).
as:images
as:documents
, as:video
, as:sound
, as:dataset
elementsas:models
Media Displays
C
Element generation from CSV)C
Element generation from CSV)JSONAPI
on a set of Drush script to (Strawberry Seeds)
We ship our deployment with a full set of configurations to get started. But sometimes, specially during development we want/you want to have some configs to be sync/able and others to be local to your own reality. Config Managment (import/export) does not allow that. So there is this
https://www.drupal.org/project/config_split
Handy module that says it can do that
There are of course old Features module, but i kinda like the idea we are not mixing things up here and are mostly relaying on core for config managment. Just too risky.
@giancarlobi wonder if you could look at this module some of these days and give me your opinion.
Config inspector, a contributed module we ship as a facility to debug missing config schemas is passing a wrong template variable and failing to render its main inspector page on Drupal 8.8.1. Moving to dev version and fixing composer.lock to a certain hash fixes the problem. This issues was discovered by @marlo-longley while testing beta2
It looks like logging in as admin user makes it impossible to see the site as a regular user, you are only exposed to admin tools. Maybe Drupal has some 'switch user' functionality that can help with this. If yes, this would be good to add to documentation
Bots are the worst. They crawl in masses and many ignore robots.txt. They over use our resources and generally end in making us having to restart services just to close open connections (cantaloupe, Solr, etc)
For archipelago (at least the default deployment) we have
https://www.nginx.com/blog/rate-limiting-nginx/
This way we can limit certain endpoints we know may suffer from rate limit issues leading to Denial of Service
@giancarlobi @dmer what we want is to have a list of possible pages (patterns) that can suffer from this
e.g
/user/logic
/search
/cantaloupe/iiif/2
/webform_strawberryfield/
/webform_strawberry/auth_autocomplete/*
/webform_strawberry/nominatim/*
/do/{node}/iiif/{uuid}/full/full/0/{format}
Real challenge is to think in terms of a user here. We want normal operations to happen correctly, we do not want to block access.
Any ideas?
we used to use Java2D for tiffs, now after the update to the latest container and in Beta2, people don't see the TIFFs (sad).
Fix is to change to Graphicsmagic and then figure out why. I have suspects, like a bad Java2D processor but also maybe the streaming strategy for TIFFS not working on 2D.
Anyway. bad, bad.
In the OSX deployment documentation, Step 4 provides two scripts for creating users "demo" and "jsonapi":
docker exec -ti esmero-php bash -c 'drush ucrt demo --password="demo"; drush urol metadata_pro "demo"'
docker exec -ti esmero-php bash -c 'drush ucrt jsonapi --password="jsonapi"; drush urol metadata_pro "jsonapi"'
I made the mistake of running both lines at the same time, which Terminal cannot complete and was only able to create "demo". I found that the user must copy each script separately and run individually in order for both the "demo" and "jsonapi" users to be created successfully.
While testing, fixing and debugging our beta releases (now) and future full releases, it would be nice to have a super simple bash script that runs composer update on anything strawberryfield (Archipelago packages) so people have a single point of interaction with a deployed Archipelago repository.
Super simple update.sh
bash script that runs composer update strawberryfield/*
and our theme of course too and, if Drupal is already installed, runs a drush cr
. Its nothing more than a facility.
Then add this to documentation.
This issue has been discussed offline and in meetings time and time again: the webform fields, their names and their purposes are not always clear.
With the development of Beta 2, let's consciously address some of these concerns and create a new Descriptive Metadata webform to be deployed as the default webform for Archipelago Digital Object (ADO) ingesting.
First things that come to mind to be updated:
@DiegoPino @marlo-longley thoughts?
Hi,
I was trying to follow the directions @ https://github.com/esmero/archipelago-deployment/blob/8.x-1.0-beta1/docs/osx.md#step-1-docker-deployment
I get
In RequestException.php line 113:
Server error: `GET https://schema.org/docs/jsonldcontext.json` resulted in a `503 Service Unavailable` response:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>503 Over Quota</title>
<style><! (truncated...)
when I run
archipelago-deployment % docker exec -ti -u www-data esmero-php bash -c "cd web;../vendor/bin/drush -y si --verbose config_installer config_installer_sync_configure_form.sync_directory=/var/www/html/config/sync/ --db-url=mysql://root:esmerodb@esmero-db/drupal8 --account-name=admin --account-pass=archipelago -r=/var/www/html/web --sites-subdir=default --notify=false install_configure_form.enable_update_status_module=NULL install_configure_form.enable_update_status_emails=NULL;drush cr;chown -R www-data:www-data sites;"
New module. An AMI dependency so we do not have to (grrmrmr ... such a bad idea, i should code all from scratch always...anyways) code GOOGLE API OAUTH2 access to Google sheets like we do on IMI. But, of course the module has bugs!
See this one i reported https://www.drupal.org/project/google_api_client/issues/3172577
The fix is like super crazy simple but it will take forever to be applied by the owner so we better do this.
Of course i will need a reminder of this by Friday =(
Yes. I spend a few hours of my free day dealing with random Docker issues. Short story is i had to change my decaying Mac for a new one and was quite happy with new speed and was coding like hell! (this happened a few weeks ago)
But I had a few strange issues with Solr indexes going to hell after a docker-compose down (assumed it was a badly made backup..) but now deploying a fresh version of 1.0.0-RC1 using Solr 8.7 and MYSQL 8.0.22 nothing worked. Lots of DB errors, constant restart and lost hope and countless
hours. (4)
Docker Desktop for Mac enables now (I have 2.5.0.1 with Docker Engine 19.03.13) by default an experimental file sharing option (same as for windows) which is not only buggy but quite buggy. Issues with async writing, lots of GitHub issues with people being annoying all that added to the fact that I was in the presence of a CASE INSENSITIVE filesystem.
gRPC FUSE
file sharing option when using bind/cached mounts. (works)All is working now (I mean testing but all starting and running). But we need to document this use case/reality/issue and also provide an alternative docker-compose file for OS X users if we do not change the /persistent/db storage via a bind mount.
GOSH!! (which is not the same as 'gosh')
NOTE:
this is the type of errors I was getting:
esmero-db | 2020-11-11T20:35:35.238363Z 1 [Warning] [MY-012637] [InnoDB] 16384 bytes should have been read. Only 0 bytes read. Retrying for the remaining bytes.
esmero-db | 2020-11-11T20:35:35.238978Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for reading partial data failed.
esmero-db | 2020-11-11T20:35:35.241630Z 1 [ERROR] [MY-012642] [InnoDB] Tried to read 16384 bytes at offset 49152, but was only able to read 0
esmero-db | 2020-11-11T20:35:35.242429Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 95 in a file operation.
esmero-db | 2020-11-11T20:35:35.242921Z 1 [ERROR] [MY-012596] [InnoDB] Error number 95 means 'Operation not supported'
esmero-db | 2020-11-11T20:35:35.244192Z 1 [ERROR] [MY-012646] [InnoDB] File ./undo_001: 'read' returned OS error 195. Cannot continue operation
esmero-db | 2020-11-11T20:35:35.246773Z 1 [ERROR] [MY-012981] [InnoDB] Cannot continue operation.
esmero-db | 2020-11-11 20:35:36+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
These 2 fields are labeled so similar, it is confusing some users.
Change the labels. Suggestion:
Object Title becomes: Object Label (to be used in the metadata)
Title becomes: Object Title (display label)
Right now when I enter an Object Title on step 1 of the webform wizard, and hit save to go to step 2, the title I entered is erased or at least not retrieved.
Not sure if others can reproduce that, or if there is some background about this I don't know yet.
A bad title for a good show off of our Twig/IIIF capabilities.
Actually this is something that came through a shared use case and my constant avoidance of harcoding things we do not want to hardcode because everyone will want a different need. Thumbnails. Yes, old fashioned ones.
We provide a simple IIIF Image formatter (and configured by default for thumbnails) that takes the first image in any ADO and renders a thumbnail using Cantaloupe. Great!. But what about a PDF and no image? Or Objects with just ZIP files where the rdf type is more important than the MEDIA itself. And Movies? Well. We can, i mean this is ARCHIPELAGO with folks coding on a friday at 21:24, so it is possible (actually since 2018) π₯°
But what if the solution instead of more specific formatters is just the existing Metadata formatter using a Twig template one?
Well, here it is! Including gloriously inline docs @giancarlobi @alliomeria @tomadams (the one that explicitly asked for the use case) .
/metadatadisplay/add
, set as HTML, Save. I named it "Many thumbnails" but you are all smarter and will give it a better name.Search results View Mode
(first, then you can also do that for the collection listing too) at/admin/structure/types/manage/digital_object/display/strawberryfield_processed_metadata
Things you can change (everything really)
pronom IDs
, like only pdf 1.3
and 1.4
but never 2.0
)Basically control, the love of the icon and display is back to you dear user.
What can go wrong. Basically same that can go wrong with unknown media, missing media, etc. Good thing is no recoding/asking for a new release is needed. Just add more test/checks in the conditionals (flv:exif
has a lot you can use to decide if processing will be even possible) and learn.
Don't forget to clear your caches after the change(s)
{# This Template generates Thumbnails for various Media representations
Structures passed from Archipelago are
- node
The full Node entitya
- data
The Full JSON Structure converted in an Array
- iiif_server
The publicly available IIIF Server (Global Setting)
Since JSON was decoded into the 'data' values are valid PHP but could not
be valid JSON when outputting. So We use data.description|json_encode|raw
Without Double quotes.
#}
{# <---- SETUP ---> #}
{# -- The Desired width -- #}
{% set width = 240 %}
{# -- The IIIF Server URLS -- #}
{% set Webserverurl = url('<front>')|render|replace({':8001/':''}) ~ ':8183/iiif/2/' %}
{% set IIIFserverurl = iiif_server ? iiif_server ~ '/': Webserverurl %}
{# -- Node Info -- #}
{% if node.id %}
{% set nodeurl = url('entity.node.canonical', {'node': node.id}, {'absolute': true}) %}
{% else %}
{% set nodeurl = '<current>' ~ '/iiif/manifest' %}
{% endif%}
{# -- Language -- #}
{% set len = language.getId() ? language.getId() : "en" %}
{# --THIS WILL BE THE IMAGE IIIF URL. CAN HAVE A DEFAULT VALUE IN CASE OF FAILURE THERE IS ALWAYS A GENERIC THUMB-- #}
{# -- NOTE: for loops are scoped so if we onlt define source inside one it will not be accesible from the outside- #}
{% set source = '' %}
{# <---- END SETUP --> #}
{# We will test for 3 main options
1.- Images
2.- PDF
3.- Movies
#}
{% apply spaceless %}
{% spaceless %}
{% if attribute(data, 'as:image')|length > 0 %}
{% set document = data['as:image']|first %}
{% set cantaloupeid = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %}
{% set source = cantaloupeid ~ "/full/" ~ width ~ ",/0/default.jpg" %}
{% elseif attribute(data, 'as:document')|length > 0 %}
{% set break = false %}
{% for document in data['as:document'] if not break %}
{# this is just an example, we allow every PDF here but we can also filter by pronom id "info:pronom/fmt/17" or info:pronom/fmt/18 #}
{% if document['flv:exif'].MIMEType == "application/pdf" and document['flv:exif'].PageCount > 0 %}
{% set cantaloupeid = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %}
{% set source = cantaloupeid ~ "/full/" ~ width ~ ",/0/default.jpg?page=1" %}
{% set break = true %}
{% endif%}
{% endfor %}
{% elseif attribute(data, 'as:video')|length > 0 %}
{% set break = false %}
{% for document in data['as:video'] if not break %}
{# this is just an example, we process here only MP4 quicktimes #}
{% if document['flv:pronom'].mimetype == "video/quicktime" %}
{% set cantaloupeid = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %}
{% set source = cantaloupeid ~ "/full/" ~ width ~ ",/0/default.jpg?time=00:00:01" %}
{% set break = true %}
{% endif %}
{% endfor %}
{% endif%}
{# <----START OUTPUT --> #}
<a href="{{ nodeurl }}" alt="{{ node.label.value }}" hreflang="{{ len }}">
{% if source|length > 0 %}
<img class="field-iiif image-iiif img-fluid rounded" id="thumb_iiif-field_descriptive_metadata-{{ node.uuid.value }}-0-image1" src="{{ source }}" width="240" alt="Thumbnail for {{ node.label.value }}" typeof="foaf:Image">
{% else %}
{# -- IF we have no source we can e.g use font awesome -- #}
{#
Options we took from /admin/structure/webform/config/options/manage/schema_org_creative_works/source
Article: Article
Book: Book
Collection: Collection
Dataset: Dataset
DigitalDocument: 'Digital Document'
Map: Map
Movie: Movie
MusicRecording: 'Music Recording'
Painting: Painting
Photograph: Photograph
Sculpture: Sculpture
VisualArtwork: VisualArtwork
Thesis: Thesis
Podcast: Podcast
PodcastEpisode: 'Podcast Episode'
Panorama: 'Panorama (http://ns.google.com/photos/1.0/panorama)'
PanoramaTour: 'Panorama Tour (http://ns.google.com/photos/1.0/panorama)'
3DModel: '3D Model'
WebPage: 'Web Page'
AudioObject: 'Audio Object'
VideoObject: 'Video Object'
Conversation: 'Conversation (Oral History)'
ShortStory: 'Short Story'
#}
{% if data.type in ['DigitalDocument', 'Article','Thesis'] %}
<i class="fas fa-file-alt fa-9x"></i>
{% elseif data.type in ['Photograph','Painting','VisualArtwork','VideoObject'] %}
<i class="fas fa-photo-video fa-9x"></i>
{% elseif data.type in ['Movie'] %}
<i class="fas fa-film fa-9x"></i>
{% elseif data.type in ['MusicRecording', 'AudioObject'] %}
<i class="fas fa-headphones-alt fa-9x"></i>
{% elseif data.type in ['Podcast','PodcastEpisode'] %}
<i class="fas fa-podcast"></i>
{% elseif data.type in ['ShortStory','Conversation',] %}
<i class="fas fa-comments fa-9x"></i>
{% elseif data.type in ['Sculpture','3DModel'] %}
<i class="fas fa-cubes fa-9x"></i>
{% elseif data.type in ['Book'] %}
<i class="fas fa-book fa-9x"></i>
{% elseif data.type in ['WebPage'] %}
<i class="fas fa-sitemap fa-9x"></i>
{% elseif data.type in ['PanoramaTour'] %}
<i class="fas fa-vr-cardboard fa-9x"></i>
{% elseif data.type in ['Dataset'] %}
<i class="fas fa-table fa-9x"></i>
{% elseif data.type in ['Map'] %}
<i class="fas fa-map-marked-alt fa-9x"></i>
{% else %}
<i class="fas file-contract fa-9x"></i>
{% endif %}
{# -- ENDIF fontawesome -- #}
{% endif %}
</a>
{% endspaceless %}
{% endapply %}
Preview here with a normal image source, a PDF and a VIDEO
Please let me know if this works or you feel it needs more logic/options. If good i will add as alternative to this deployment. Also feel free to ask about the Twig statements and things i did here if something is unclear. I tried to add as many inline comments as possible.
Good night
As good, nice, people start to deploy archipelago on production, or at least, public facing instances, having an base AWS deployment strategy that involves a real domain name and SSL using docker and hopefully AWS S3 (for a cheaper experience than super fast min.io) is crucial.
When deploying an Archipelago from scratch, the new Archipelago base theme does not render right away.
This appears to be a known issue with the color module: https://www.drupal.org/project/drupal/issues/2995825
Quick fix for Archipelago users is to go to /admin/appearance/settings/archipelago_subtheme
, click Save
at the very bottom, clear your cache and refresh. Theme should then load and be good to go!
Switch View Mode is used to allow to decide how a DIGITAL Objects is going to be displayed, in other words that Field Formatter combo, Also known as Display Mode will apply over the metadata. The default one is always a IIIF Viewer using OpenSeadragon.
This 1660619
restores that functionality by default. If you have that functionality missing, please go to "Extend"
and enable the Display Suite Switch View Mode submodule
as seen here:
Alternatively, you can do a git pull and sync the configurations again at http://localhost:8001/admin/config/development/configuration
which gives you also a good chance to learn how Drupal manages configurations, live and stored on file.
See also #5
This is our working enumeration of concrete tasks for this year, per Component and Service, for public evaluation (critics and comments welcome).
All tasks listed here are leading our full feature release somewhere this first half of the year. We went more ambitious (we planned for end of 2019) so code has been moving along and we feel we are quite there! Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.
Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).
as:images
as:documents
, as:video
, as:sound
, as:dataset
elementsas:models
Media Displays
JSONAPI
on a set of Drush script to (Strawberry Seeds)
~/Development/archipelago-deployment $ docker-compose up -d
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?Supported filenames: docker-compose.yml, docker-compose.yaml
I renamed the file (mv docker-compose-nginx.yml docker-compose.yml
) and everything continues as expected.
Ok, not everybody but I just found myself doing it and it was not fun.
Changes we need to make this happen (this is a mix of documentation but also testing new things)
- ${PWD}/persistent/ngixncache:/var/cache/nginx:cached
We may want multipart upload too. With resuming capabilities: there are a number of JS libraries that can handle this but I also like the nginx native approach, here:
@giancarlobi you use Apache I know, but this is needed so maybe you have backend independent suggestions we may want to add
Also:
All this implies changes to our docker deployment. Right now we are using the generic Nginx docker container but also, for SSL I moved to staticfloat/nginx-certbot which includes not only base system but also automatic renewal and request of SSL certs (free ones, fully valid) and non interactive which means just a few more seconds startup time but no copying .well-known hashes around and sending prayers. I feel if we are going this route, we may need to extend our own Docker Container for nginx.
if you are using like myself Archipelago deployment to develop and you are putting a lot of data out, changes are that nginx will complain and give you a few 502 errors. Reason, buffer is just too small for Development (when you are outputting JSON and stuff out).
Add
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
To nginxconfigford8/default.conf
, restart and all good!
Pull coming!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.