indico / indico-migrate Goto Github PK

• Chat
• Invenio Importer - just settings
• LiveSync - if we ignore it people need to add their agent manually and either run an initial export or set it as run via indico shell
• PayPal Payment - important event settings such as the recipient of the payment
• Piwik - just settings
• Search Invenio - just settings
• VC Vidyo

• event_data
• event_alarms
• event_logs
• event_notes
• event_images
• event_stubs
• event_menus
• event_layout
• event_surveys
• event_managers
• event_regforms
• event_participants
• event_timetable
• event_locations
• event_categories
• event_dates_titles
• event_abstracts_zodb / event_abstracts
• event_paper_reviewing_legacy / event_papers
• event_acls
• event_types
• event_keywords_ctimes_visibility
• event_shorturls

• users
• groups

Should be split between:

• Event migration
• Category migration

After attachments.

See indico/indico#3066

This probably needs to be changed not only in _migrate_registration_choice_field but also in _migrate_registration_statuses

I have also a number of events where indico-migrate reported a timetable error like:

timetable   Contribution <persistent broken MaKaC.conference.Contribution instance '\x00\x00\x00\x00\x00\x16\x8a#'> has no fields

An event with such an error in my database is a0681. I have no clue what the potential impact on the converted event could be.

I did a check to compare the occurence of this error with the layout error and I found that it is often but not always associated with a layout error. The attached file is the result of the check done with the following command ($events containing all the event ids where a timetable error occured):

for event in $events; do echo Checking layout errors for $event; grep layout migration.log|grep " $event "; done>/tmp/timetable.list

From the visual checks I made, it seems to be again CDSAgenda events. And as usual you can look at these events in more details at http://indico.lal.in2p3.fr (v1.2).

I think that everything is in the title. The [event] message about naive datetime is not associated with an event so it is difficult to check the result. For example:

! [event] Naive datetime converted (2006-11-24 18:00:00)

indico-migrate on our 1.2 db reported a non negligible number of layout problems with the following message (for example event a0698 in our db):

layout      Skipping some settings, missing _styleMngr attribute

Looking in more details, it appeared that the events causing this problem are coming from CDSAgenda (said explicitly in management mode) and have the layout shown in the attached screenshot (the traditional CDSAgenda look if I remember properly). Their "default style" setting is 'CDSAgenda style'. Those events are created in v2 but as a conference rather than a meeting (despite the event type in 1.2 is meeting) and I didn't find how to change it. Also the timetable style is the default one rather than the "indico style" which is very close to the CDSAgenda one (only the background customization in the description is not done).

Even if nobody should use the CDSAgenda layout for a new event, for a migrated one the layout should be as close as possible (if not identical) to what it was in 1.2 (the description yellow background is among the details that can be ignored IMO).

In our 1.2 database, we have several invalid emails entered by participants who registered to events, in particular missing @dom.ain. Currently the migration script is adding @example.com which is not really making sense IMO. Could it be possible to have an option to delete such emails rather than modifying them (keeping the participants)?

• legacy_categories
• categories
• User.favorite_categories

• Migrate rooms/locations (beginning)
• Migrate reservations (after events)

indico-migrate reported 710 events without a category in our 1.2 DB. From the tests I made these events are already unreachable in v1.2 so this is not in itself a show stopper. But looking at the problem in 1.2 when trying to access them, the error reported is:

astimezone() cannot be applied to a naive datetime

which suggests that event is there but broken. The result of the conversion is that the event is not migrated. Is there a chance to fix the problem with the "naive time" and to resurrect these events without too much efforts? It seems that the affected events all come from CDSAgenda.

BTW, if the problem really is that the event has no category in 1.2 (probably due to some problem during the CDSAgenda -> Indico conversion), would it be possible to have an option in indico-migrate to create a Lost+Found category and assign these events to it (the category name is not that important, we can change it afterward).

If you want to look in more details to such an event, use the following URL: https://indico.lal.in2p3.fr/event/LAPPa079/.

Looking with indico_shell to the event in 1.2 gives the following result:

>>> pp.pprint(ConferenceHolder().getById('LAPPa079').__dict__)
{   '_Conference___contribTypeGenerator': <MaKaC.common.Counter.Counter object at 0x737fb90>,
    '_Conference__ac': <MaKaC.accessControl.AccessController object at 0x737fb18>,
    '_Conference__alarmCounter': <MaKaC.common.Counter.Counter object at 0x73812a8>,
    '_Conference__contribGenerator': <MaKaC.common.Counter.Counter object at 0x737fe60>,
    '_Conference__creator': <Avatar(1, CDS Agenda)>,
    '_Conference__materialGenerator': <MaKaC.common.Counter.Counter object at 0x737fc80>,
    '_Conference__owners': [],
    '_Conference__programGenerator': <MaKaC.common.Counter.Counter object at 0x737f668>,
    '_Conference__schedule': <MaKaC.schedule.ConferenceSchedule object at 0x737f8c0>,
    '_Conference__sessionGenerator': <MaKaC.common.Counter.Counter object at 0x737fcf8>,
    '_IM2Categs': [],
    '_OAImodificationDS': datetime.datetime(2007, 2, 5, 20, 27, 7, 338606),
    '_accessKey': '',
    '_archiveId': None,
    '_archivingDate': None,
    '_archivingRequestDate': None,
    '_authorIdx': <MaKaC.conference.AuthorIndex object at 0x737faa0>,
    '_autoSolveConflict': True,
    '_boa': <MaKaC.conference.BOAConfig object at 0x7381230>,
    '_bookings': {   },
    '_chairGen': <MaKaC.common.Counter.Counter object at 0x7381140>,
    '_chairs': [<MaKaC.conference.ConferenceChair object at 0x73810c8>],
    '_closed': False,
    '_contribTypes': {   },
    '_creationDS': datetime.datetime(2007, 1, 19, 0, 0),
    '_enableSessionSlots': True,
    '_legacyAlarmList': [],
    '_logHandler': <MaKaC.common.log.LogHandler object at 0x737fed8>,
    '_logo': None,
    '_modifKey': '41304100',
    '_modificationDS': datetime.datetime(2013, 5, 30, 11, 1, 27, 789721, tzinfo=<UTC>),
    '_participation': <MaKaC.participant.Participation object at 0x737f7d0>,
    '_pendingQueuesMgr': <MaKaC.common.pendingQueues.ConfPendingQueuesMgr object at 0x737f6e0>,
    '_primAuthIdx': <MaKaC.conference._PrimAuthIdx object at 0x7381050>,
    '_registrantGenerator': <MaKaC.common.Counter.Counter object at 0x737f758>,
    '_registrants': {   },
    '_registrationForm': <MaKaC.registration.RegistrationForm object at 0x737f9b0>,
    '_reportNumberHolder': <MaKaC.conference.ReportNumberHolder object at 0x737fde8>,
    '_sections': [],
    '_sessionCoordinatorRights': [],
    '_sessionCoordinators': <MaKaC.conference.SCIndex object at 0x737fc08>,
    '_speakerIdx': <MaKaC.conference.AuthorIndex object at 0x737f5f0>,
    '_submissionDate': None,
    '_submitterIdx': <MaKaC.conference.SubmitterIndex object at 0x737f938>,
    '_supportInfo': <indico.core.db.event.SupportInfo object at 0x73811b8>,
    '_trackCoordinators': <MaKaC.conference.TCIndex object at 0x737fd70>,
    '_visibility': 999,
    'abstractMgr': <MaKaC.review.AbstractMgr object at 0x737fa28>,
    'alarmList': {   },
    'chairmanText': '',
    'chairmans': [],
    'contactInfo': '',
    'contributions': {   's1t0': <MaKaC.conference.Contribution object at 0x737f848>},
    'description': '',
    'endDate': datetime.datetime(2007, 1, 19, 17, 0),
    'id': 'LAPPa079',
    'materials': {   },
    'minutes': None,
    'paper': None,
    'places': [<MaKaC.conference.CustomLocation object at 0x7381320>],
    'poster': None,
    'program': [],
    'programDescription': '',
    'rooms': [],
    'sessions': {   },
    'slides': None,
    'startDate': datetime.datetime(2007, 1, 19, 15, 30),
    'title': 'r\xc3\xa9union LHCb',
    'video': None}

Should be fixed in the migration script.

SQL command to fix it in an already-migrated DB:

SELECT setval(pg_get_serial_sequence('events.events', 'id'), (SELECT max(id) FROM events.events));

Attachment migration should be split between:

• Event migration
• Category migration

Good Morning,

I am getting the following error when trying to migrate fom 1.2 to 2.0.:

(psycopg2.IntegrityError) null value in column "storage_backend" violates not-null constraint
DETAIL:  Failing row contains (1, 109, 1, 0, null, application/pdf, , null, senekane_et_al_dp_student_t_mechanism.pdf).
 [SQL: 'INSERT INTO event_paper_reviewing.files (contribution_id, revision_id, size, storage_backend, content_type, md5, storage_file_id, filename) VALUES (%(contribution_id)s, %(revision_id)s, %(size)s, %(storage_backend)s, %(content_type)s, %(md5)s, %(storage_file_id)s, %(filename)s) RETURNING event_paper_reviewing.files.id'] [parameters: {'storage_file_id': None, 'filename': 'senekane_et_al_dp_student_t_mechanism.pdf', 'contribution_id': 109, 'storage_backend': None, 'content_type': 'application/pdf', 'revision_id': 1, 'md5': u'', 'size': 0}]

Hi,

I am getting the following errors when doing an indico Migration:

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/local/lib/python2.7/site-packages/indico_migrate/steps/events/misc.py:109 [migrate]
 if not alarm.startDateTime:

'Alarm' object has no attribute 'startDateTime'