beit-hatfutsot / beit-hatfutsot-devops Goto Github PK

We are currently looking for the following profiles of developers (from which we won't necessarily recruit all 3 profiles, we might recruit less or more) :

• Junior backend (Python / Elasticsearch)
• Junior frontend (Html5 / Css / Js / Angular)
• Senior fullstack

including, but not limited to:

• databases app mongo / elasticsearch
• IDF app mongo
• ?

related issues

• Beit-Hatfutsot/dbs-back#123

reproduction steps

• make changes in DNS in Cloudflare (they manage our DNS)
• for example, I changed devapi.dbs.bh.org.il to point to IP 104.155.73.237
• wait for DNS change to take effect (should be fast, but I waited more then 48 hours to be sure)

expected

• DNS takes effect everywhere

actual

• users from internal Beit Hatfutsot network don't reach the correct IP
• all other external users reach the correct IP and see the DNS change

browser stack gives free service for open source, need to contact them here: https://www.browserstack.com/contact?ref=open_source#open-source

Currently there are updates to content on BHP that we do not yet display.
We need to run migration manually for all non-ftree items we do not yet have the latest versions for.

Verify:
http://dbs.bh.org.il/he/%D7%A6%D7%A8%D7%A0%D7%99-%D7%9C%D7%90%D7%95%D7%9F-%D7%94%D7%90%D7%A8%D7%99%D7%A1/%D7%90%D7%99%D7%A9%D7%99%D7%95%D7%AA

צ'רני, לאון האריס
will be replaced with
צ'ארניי, לאון האריס

@OriHoch commented on Tue May 23 2017

see - https://app.codeship.com/projects/175641/builds/25363392?pipeline=89420f43-6f7d-4bfb-be97-0f3a6a7346ad

need to prevent this from happening, and ensure disk space never runs out..

@nuritgazit commented on Sun Apr 23 2017

High level design is here
Design is here
Definition is here

@nuritgazit commented on Mon Apr 24 2017

@daonb please add a link to the document you created

@daonb commented on Sun Apr 30 2017

Added it at the top of the description

@OriHoch commented on Mon May 01 2017

needs tech design - ~2-3 points

Added a team called "active-contributors" to replace the old team "devs"

This new team gives write access to it's members for all relevant BH repositories

It is intended for all contributors (whether it's code contributions or product/project management contributions)

It can also be used to mention the team in issue comments, like so:

@Beit-Hatfutsot/active-contributors

reproduction steps

• be a developer that needs to work on wikipedia integration
• read the details in #19

expected

• should have details about which pages we should import from wikipedia to be displayed on BH
• assuming we need only "Jewish" wikipedia articles - how do we know which ones?

actual

• there are details about how a specific wikipedia item page will look
• there are details about how to import a specific wikipedia page to BH
• no details about which pages to import / how to decide which pages to import

@nuritgazit commented on Wed Jun 07 2017

We are in contact with different institutions who do not have internal developers / resources to develop an API, but still want to collaborate. They need our guidance / technical support / technical solution, in order to allow us to integrate their content on our unified search.

More details are here in this board: https://trello.com/b/nL3Wr7Ou/the-ring

core tasks

• setup the pipelines framework and sync of basic data (content, title) from CM (Clearmash) to ES (Elasticsearch)
• adjust backend to use the data from the new ES
• adjust frontend to match the backend changes (mostly schema changes in returned data)
• adjust the pipelines sync to add more details to ES
• adjust the pipelines sync to support more sync features (e.g. updating / deleting items based on permissions)

supporting tasks

Additional tasks which can help mitigate risks and allow for more efficient development of the core tasks

• test the data in ES
• review all existing frontend / backend high-level features to determine what will be affected

issues

All the clearmash issues are labeled clearmash you can see them all across all the repos here:

• https://github.com/issues?utf8=%E2%9C%93&q=repo%3Abeit-hatfutsot%2Fdbs-back+repo%3Abeit-hatfutsot%2Fdbs-front+repo%3Abeit-hatfutsot%2Fmojp-dbs-pipelines+repo%3Abeit-hatfutsot%2Fbeit-hatfutsot-devops+label%3Aclearmash

Summarize all the requests we got from editors and users that were handled by a sysadmin and write a guide covering the common scenarios.

Sent by Nurit Gazit ([email protected]). Created by fire.

From: Oribi [mailto:[email protected]]
Sent: Monday, May 1, 2017 10:49 AM
To: Nurit Gazit <[email protected]>
Subject: Welcome to Oribi

Hi and welcome on board,

To get started with Oribi, add the tracking code to your site. It's super simple and should only take 1-minute. This is the first and last time you'll be asked to use code.

Get your unique tracking code here: https://oribi.io/get-tracking-code/Xy04ODQzNTA5MzE

Like most tracking codes, it needs to be added just before the closing </head> tag on all pages.

If something isn't clear, or you just feel you'd like to learn more about Oribi, simply reply to this email to schedule a live demo with one of our (super nice) success managers.

=========

In any case, here are some quick links that might be relevant:

• Google Tag Manager - Add the tracking code to Google Tag Manager by using the Custom HTML tag type

• WordPress - in case you don't have one installed already, there are plenty of free WordPress plugins that will allow you to easily insert external code. A good place to look for a plugin is the WordPress Plugins Directory

• Unbounce - Add the code to the Javascripts section

Thanks,
Team Oribi

Questions? We're here! Just reply to this email

Greetings, fire here!

@OriHoch recently invited @fire-bot to this repository. Before fire is enabled, @OriHoch needs to complete a few steps:

Complete fire setup now

Only @OriHoch will be able to enable fire using the above link. If someone added fire by mistake, feel free to remove @fire-bot from your repo's collaborators.

Sent by @OriHoch ([email protected]). Created by fire.

עץ 12209 - באונליין לא מוצג העץ הנכון. העץ הנכון הוא לא עץ בעברית של 15 שמות, אלא עץ בלועזית עם 4538 שמות.____

http://dbs.bh.org.il/person?tree_number=12209&more=1____

__

Attachments:

1. image001-18.jpg
2. image001.jpg

@nuritgazit commented on Tue May 16 2017

the item 26559 from BHP, that has no internet rights, and does not appear on the DBs site (which is good), does appear in search results on CJH (which is not good)

reproduction

• search for DARI, DERI, DER'I, DEREHA, EDRY, EDERY, EDREHY
  • you can use this prepared search on Kibana
• results contain 7 results of different family names all based on DERHI variants
• look at the content (unit text) of those family names

expected

• no duplicated content

actual

• content is the same for all the 7 family names

implications of this bug

• while this might not look like a bug - it could have consequences on the search enging
• it might skew results and prevent the search engine from determining relevancy properly
• need to research this problem to determine if it really is a problem and what can be done about it

TODO

• look for more examples of this problem - are there more family names which have duplicated content?
• do we have duplicated content in other collections?
• research duplicated content in elasticsearch and how we can deal with it

Interim report 2.xlsx

The middle tab of the attached document contains URLs that produce images that return 404.
We need to figure out why this happens.

specific tree (according to tree number) should not be displayed (according to BHP permissions)

need to remove this tree from the online site ASAP

TODO

1. get access to the private repo
2. review the code and remove sensitive details
3. publish to GitHub

@nuritgazit [9:41 AM]
I can't seem so search the site
gets the "loading" icon
Tried "strimban" and got nothing
tried "Warsaw" and got the CJH and Europeana in the background
but still the loading icon on top

@Libisch [9:43 AM]
(links to specific items work)

@nuritgazit [9:52 AM]
looks like wizard suggestions aren't working as well

@TheGrandVizier [9:52 AM]
I think api is down

באונליין מופיעה הודעה צמודה לפריט זה שעל פיה ישנן לכאורה עוד שתי תמונות אשר עדיין לא עברו דיגיטציה. זאת הודעה שגויה, לפריט זה יש רק סריקה אחת שאותה אכן רואים באתר. ____

http://dbs.bh.org.il/image/facade-of-the-moked-house-and-strashon-library-vilna-1947____

http://dbs.bh.org.il/he/%D7%97%D7%96%D7%99%D7%AA-%D7%94%D7%91%D7%A0%D7%99%D7%99%D7%9F-%D7%91%D7%95-%D7%94%D7%99%D7%94-%D7%91%D7%99%D7%AA-%D7%94%D7%9E%D7%95%D7%A7%D7%93-%D7%95%D7%94%D7%A1%D7%A4%D7%A8%D7%99%D7%94-%D7%A2%D7%9C-%D7%A9%D7%9D-%D7%A1%D7%98%D7%A8%D7%A9%D7%95%D7%9F-%D7%95%D7%99%D7%9C%D7%A0%D7%94-1947/%D7%AA%D7%9E%D7%95%D7%A0%D7%94____

@nuritgazit commented on Thu May 04 2017

We need a technical document that describes the 2 ways we can collaborate with different institutions in the DBs - rest API / Data Dump.
Using this document, Ronen's team can approach potential collaborators and instruct them on what to do so that we can integrate their content on our search.
@daonb, I think you are the natural candidate for that...?

currently the app code is in a private repository, need to open source it

TODO

• get the private code
• remove sensitive details
• publish to GitHub
• add some documentation

In order to have the ability to control the instagram computer from the office, we need to set up a remote desktop connection.
Windows has remote desktop built in, so only setup is required.

@nuritgazit we should be able to do this without involving Mr Madmon.

reproduction steps

• open IDF trees web-app
• register and add your tree details
• wait for beit hatfutsot to get the update

expected

• should get updated summary periodically (e.g. every week)

actual

• need to ssh into the idf trees server and run the following:

sudo su -l ftapp
cd /home/ftapp/current/
python send-trees.py -d gentreeDb -e '[email protected]'

notes

• it's easiest to send all data every time (it's not too big) but if needed we can send only since last summary was sent

כמו כן, עדיין קיימת בדוחות אלה בעיה שאני מדבר עליה כבר שנתיים, וללא תיקונה אי אפשר לעבוד בצורה יעילה. אי אפשר לדעת כמה רשומות קיימות בקובץ, הדיווח לא צריך להיות על "new persons", אלא על סה"כ רשומות בכל גרסה של הקובץ.

להלן לינק אל עץ # 12189 אשר לא מוצג באונליין:

http://dbs.bh.org.il/person?first_name=%D7%A8%D7%95%D7%A0%D7%9F&amp;last_name=%D7%93%D7%A0%D7%99%D7%90%D7%9C&amp;tree_number=12189&amp;more=1

Sent by @OriHoch ([email protected]). Created by fire.

---------- Forwarded message ---------
From: Ori Hoch <[email protected]>
Date: Sun, Mar 26, 2017 at 9:38 AM
‪Subject: Fwd: תקלה באונליין‬
To: <[email protected]>

עץ 12209 - באונליין לא מוצג העץ הנכון. העץ הנכון הוא לא עץ בעברית של 15 שמות, אלא עץ בלועזית עם 4538 שמות.____

http://dbs.bh.org.il/person?tree_number=12209&more=1____

__

Attachments:

1. image001-16.jpg
2. image001.jpg

should have a proxy available e.g. at https://dbs.bh.org.il/cjh/
which will act as a proxy and route all requests to CJH and back

this will allow to embed the CJH search results in our frontend site