The idea is to add the domain to all method signatures so that clients/conveyor can specify which domain should be used to create protocols/pathes.

• First iteration: Add domain to method signature, without implementing any functionality yet.
• Second iteration: The right output is generated depending on domain input. If no domain is given, wan is used, if wan does not exist, lan is used
• Third iteration: Client / Conveyor have to call the methods with the right domain.

Migrated from JIRA RUCIO-2778

Migrated from JIRA RUCIO-2418

Set the expiration date of the rule to 4h (~) in the future if the child_rule is not OK yet, so that there is no jam of rules in front of the queue.

Migrated from JIRA RUCIO-2770

Migrated from JIRA RUCIO-2337

Migrated from JIRA RUCIO-2669

Right now, these stuck rules are repaired in Hard mode, meaning all locks are selected, while it is actually enough to just select all STUCK locks and their sister locks.

Migrated from JIRA RUCIO-2756

We need a new submission script for pull-requests on github.

Add a flag --extract, which will automatically extract compressed files.

To be precise: This should download a full zip file and extract a specific file needed from that archive.

Migrated from JIRA RUCIO-2633

test!

Migrated from JIRA RUCIO-2616

Migrated from JIRA RUCIO-2611

ntplib
etc

Migrated from JIRA RUCIO-2629

Migrated from JIRA RUCIO-2764

Migrated from JIRA RUCIO-2685

Migrated from JIRA RUCIO-2706

If the src_rse_id is pre-filled the throttler can release requests based on links, and not only based on destination.

Migrated from JIRA RUCIO-2773

The documentation on the CLI http://rucio.cern.ch/client_howto.html needs to be updated, e.g. :

Download specific files from a dataset

This operation is still not supported by rucio, but will be available soon.

Download a sample of n random files from a dataset

This operation is still not supported by rucio, but will be available soon.

Migrated from JIRA RUCIO-2498

Hermes does not protect against double selection of messages, specifically in a phase where the partitions are not well established yet, e.g. during a restart.
It does retrieve the messages with a SELECT FOR UPDATE, but the retrieval only takes ms and after that the row locks are given up. For it to be effective the locks have to be held until the message is removed from the table. This might not be a good idea to do.
One possible solution would be to have an additional state column and the retrieve_message function SELECTS the messages (with for update) and updates their state to SUBMITTING. In subsequent retrieve calls, only messages in state WAITING or which are in the SUBMITTING state for more than 10min are selected.
To be discussed.

Migrated from JIRA RUCIO-2722

• Poller
• Submitter

https://pypi.python.org/pypi/threadpool

Migrated from JIRA RUCIO-2785

• Decide on Message format for ActiveMQ Message to AMI: #70
• Schema change for database to add Partially Available(Imcomplete) to state column
• Setting dataset/container state to PA and send message to AMI when first file gets deleted in Dataset #71

Migrated from JIRA RUCIO-2809

Migrated from JIRA RUCIO-2824

Migrated from JIRA RUCIO-2812

Migrated from JIRA RUCIO-2349

Migrated from JIRA RUCIO-2247

Migrated from JIRA RUCIO-2761

Migrated from JIRA RUCIO-2819

It would be very convenient for this command to declare all replicas of the files lost:

[dcameron@lxplus078] ~> rucio-admin replicas declare-bad --reason "bad guid metadata" mc15_13TeV:HITS.10672838._007007.pool.root.1 mc15_13TeV:HITS.10672838._007080.pool.root.1
No JSON object could be decoded
Rucio exited with an unexpected/unknown error, please provide the traceback below to the developers.
Traceback (most recent call last):
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.10.1/bin/rucio-admin", line 107, in new_funct
return function(*args, **kwargs)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.10.1/bin/rucio-admin", line 730, in declare_bad_file_replicas
non_declared = client.declare_bad_file_replicas(pfns=chunk, reason=args.reason)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.10.1/lib/python2.7/site-packages/rucio/client/replicaclient.py", line 43, in declare_bad_file_replicas
exc_cls, exc_msg = self._get_exception(headers=r.headers, status_code=r.status_code, data=r.content)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.10.1/lib/python2.7/site-packages/rucio/client/baseclient.py", line 191, in _get_exception
data = parse_response(data)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.10.1/lib/python2.7/site-packages/rucio/common/utils.py", line 193, in parse_response
return json.loads(data, object_hook=datetime_parser)
File "/usr/lib64/python2.6/json/_init_.py", line 318, in loads
return cls(encoding=encoding, **kw).decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

At least a proper error message should be returned instead of a stacktrace.

Migrated from JIRA RUCIO-2622

Migrated from JIRA RUCIO-2615

Tracking ticket for Py3 migration

• Clients: #819
• Daemons: #1924
• Server:
  • REST API: #1738
  • Core: #1912

Existed in PandaQueues as WAN sink limits. We need a new way to do this in DDM.

Migrated from JIRA RUCIO-2804

Hello,
I'm the current DAST shifter and a user reported that hitting the 'Check quota' button on R2D2 hangs forever.
I checked with firefox 52.0.2 on ubuntu 16.04 and I have the same problem. The debug console points to a java script error: "option.errors s not a function".
Laurent

Migrated from JIRA RUCIO-2627

Migrated from JIRA RUCIO-2813

Migrated from JIRA RUCIO-2477

When inserting messages to hermes which are too long this throws a DB exception

DatabaseException: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (cx_Oracle.DatabaseError) ORA-01461: can bind a LONG value only for insert into a LONG column [SQL: 'INSERT INTO atlas_rucio.messages (id, event_type, payload, updated_at, created_at) VALUES (:id, :event_type, :payload, :updated_at, :created_at)'] [parameters: {'id': , 'created_at': datetime.datetime(2017, 5, 18, 14, 26, 32, 316246), 'event_type': 'email', 'updated_at': datetime.datetime(2017, 5, 18, 14, 26, 32, 316240), 'payload': '{"body": "A new rule has been requested for approval in Rucio.\\n\\nRule description:\\n\\n ID: 89593c3ea1e449cea3b2dbbd81033b06\\ ... (3749 characters truncated) ... TED MESSAG\\n", "to": ["[email protected]"], "subject": "[RUCIO] Request to approve replication rule 89593c3ea1e449cea3b2dbbd81033b06"}'}]

We have to look into if we can update the DB to allow longer messages, but in anyway, hermes should protect against this.

Migrated from JIRA RUCIO-2662

When the latest replica (eligible for deletion) is removed, the file is removed from the content but not archived.

Migrated from JIRA RUCIO-2570

Migrated from JIRA RUCIO-2704

4:24 PM
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/rucio/rse/protocols/s3boto.py", line 180, in connect
calling_format=OrdinaryCallingFormat())
File "/usr/lib/python2.7/site-packages/boto/init.py", line 141, in connect_s3
return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 191, in init
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in init
host, config, self.provider, self._required_auth_capability())
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 987, in get_auth_handler
ready_handlers.append(handler(host, config, provider))
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 144, in init
HmacKeys.init(self, host, config, provider)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 81, in init
self.update_provider(provider)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 148, in update_provider
super(HmacAuthV1Handler, self).update_provider(provider)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 85, in update_provider
self._hmac = hmac.new(self._provider.secret_key.encode('utf-8'),
AttributeError: 'tuple' object has no attribute 'encode'

Migrated from JIRA RUCIO-2657

Migrated from JIRA RUCIO-2817

In the code it seems like this is not properly implemented.

Migrated from JIRA RUCIO-2794

Do we need two different messages? One for Dataset one for Container or is the same one sufficient?

Migrated from JIRA RUCIO-2810

We need a protection for deleting a RSE. In particular, it should fail if some rules, collection replicas exist.

Migrated from JIRA RUCIO-2103

Migrated from JIRA RUCIO-2593

Migrated from JIRA RUCIO-2703

Migrated from JIRA RUCIO-2625

Also include some information about how to add a new transfertool

Migrated from JIRA RUCIO-2512

Migrated from JIRA RUCIO-2795

Migrated from JIRA RUCIO-2677

Motivation

For local usage, a combination of wan/lan attributes with rse_expressions will provide the local or remote replicas to a site.

Modification

added lan/wan in list_replicas (clients/rest/server). default wan

Based on fts RSE attribute :

fts: https://fts.usatlas.bnl.gov:8446,https://fts3.cern.ch:8446,https://fts3-test.gridpp.rl.ac.uk:8446

Migrated from JIRA RUCIO-2684

Lots of sites already support WebDAV downloads, so it would be a nice feature if we would have download directly from the WebUI. So when users list the files in a dataset their can just click a button and the files / one file would be directly downloaded to their machine.

Migrated from JIRA RUCIO-1630