Skip to content

Option to cache input data cannot be used with XRootD prefixes #21

@vepadulano

Description

@vepadulano

The function used to cache the input files on user's request only works with https based paths

def cache_files(file_paths: list, cache_dir: str, remote_prefix: str):
for url in file_paths:
out_path = Path(cache_dir) / url.removeprefix(remote_prefix).lstrip('/')
out_path.parent.mkdir(parents=True, exist_ok=True)
if not out_path.exists():
with tqdm(unit='B', unit_scale=True, unit_divisor=1024, miniters=1, desc=out_path.name) as t:
urlretrieve(url, out_path.absolute(), reporthook=_tqdm_urlretrieve_hook(t))

We could support also caching files in case XRootD paths are used

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions