Files
galaxy/test/functional/tools/data_manager_bundle_chain.xml
mvdbeek 7a585cc139 Use the bundle index for chained data-manager metadata
The mOTUs -> SameStr data-manager workflow in galaxyproject/idc#89 still
fails on fresh builds on test.galaxyproject.org. On September 17, 2026,
mOTUs 3.1.0+galaxy2 produced a bundle whose index contained the correct
relative database path, but whose HDA metadata retained the absolute
compute-job path. SameStr 1.2025.111+galaxy4 rendered that stale
metadata path and failed to copy the database, although its files were
accessible through the dataset API. Both jobs were in bundle mode and
neither was copied from a cached job; this is not an old-cache or
missing-mode issue.

PR #23097 made bundle-index paths relative and identified path columns
from the table definition. PR #23120 reconstructed the compute-side
output root, including UUID-named job outputs on an ID-keyed object
store. Those fixes are working: the live bundle index is normalized. PRs
for bare-dict rows, as emitted by mOTUs. SameStr +galaxy4 also removed
its incorrect extra /db_mOTU component. None of these changes made HDA
metadata use the normalized bundle index, so the downstream path
remained wrong.

Data-manager metadata is initially collected before bundle writing and
reads the primary JSON, which can contain absolute compute-side paths.
Chained consumers read that metadata rather than the normalized index.
Read bundle metadata from _gx_data_bundle_index.json in
DataManagerJson.set_meta and refresh metadata after the index is
written. Changing set_meta alone would not repair the initial metadata
because the index does not yet exist during that first pass. Subsequent
metadata recomputation now uses the same normalized source.

Ordinary data managers continue to read the primary JSON, and the
primary output and tool-hook ordering remain unchanged. Resolve consumer
paths on copies of the metadata rows so repeated reads cannot persist
stale absolute paths.

Add a local producer-to-consumer integration regression using Galaxy's
data-manager JSON protocol, with bare-dict/no-dbkey and list rows. It
verifies the absolute producer path, relative index and metadata paths,
metadata recomputation, repeated reads without mutation, and successful
downstream workflow consumption from the persisted dataset directory.
The test uses no mocks, downloads, or containers.

Validation: the regression fails against the pre-fix code at the metadata/index mismatch. Both chaining cases and the existing populate-mode integration test pass. Focused bundle/import, dynamic-options, datatype, job-wrapper, finishing-state, and expression-tool tests: 104 passed, 1 skipped. tox -e lint,format,mypy passes, including mypy over 2392 source files. These are local results; the change has not been deployed or verified on the live service.
2026-09-18 16:06:16 +02:00

30 lines
1.1 KiB
XML

<tool id="data_manager_bundle_chain" name="Bundle chaining fixture" tool_type="manage_data" version="1.0" profile="24.0">
<command detect_errors="exit_code"><![CDATA[
python '$__tool_directory__/data_manager_bundle_chain.py' '$out_file'
#if $source.kind == 'bundle'
--database '$source.database.fields.path'
#end if
]]></command>
<inputs>
<param name="shape" type="select">
<option value="dict">Bare dict without dbkey</option>
<option value="list">List with dbkey</option>
</param>
<conditional name="source">
<param name="kind" type="select">
<option value="new">Create database</option>
<option value="bundle">Read database from bundle</option>
</param>
<when value="new" />
<when value="bundle">
<param name="database" type="select">
<options from_data_table="testbeta" />
</param>
</when>
</conditional>
</inputs>
<outputs>
<data name="out_file" format="data_manager_json" />
</outputs>
</tool>