mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-11-22 23:10:20 +08:00
The mOTUs -> SameStr data-manager workflow in galaxyproject/idc#89 still fails on fresh builds on test.galaxyproject.org. On September 17, 2026, mOTUs 3.1.0+galaxy2 produced a bundle whose index contained the correct relative database path, but whose HDA metadata retained the absolute compute-job path. SameStr 1.2025.111+galaxy4 rendered that stale metadata path and failed to copy the database, although its files were accessible through the dataset API. Both jobs were in bundle mode and neither was copied from a cached job; this is not an old-cache or missing-mode issue. PR #23097 made bundle-index paths relative and identified path columns from the table definition. PR #23120 reconstructed the compute-side output root, including UUID-named job outputs on an ID-keyed object store. Those fixes are working: the live bundle index is normalized. PRs for bare-dict rows, as emitted by mOTUs. SameStr +galaxy4 also removed its incorrect extra /db_mOTU component. None of these changes made HDA metadata use the normalized bundle index, so the downstream path remained wrong. Data-manager metadata is initially collected before bundle writing and reads the primary JSON, which can contain absolute compute-side paths. Chained consumers read that metadata rather than the normalized index. Read bundle metadata from _gx_data_bundle_index.json in DataManagerJson.set_meta and refresh metadata after the index is written. Changing set_meta alone would not repair the initial metadata because the index does not yet exist during that first pass. Subsequent metadata recomputation now uses the same normalized source. Ordinary data managers continue to read the primary JSON, and the primary output and tool-hook ordering remain unchanged. Resolve consumer paths on copies of the metadata rows so repeated reads cannot persist stale absolute paths. Add a local producer-to-consumer integration regression using Galaxy's data-manager JSON protocol, with bare-dict/no-dbkey and list rows. It verifies the absolute producer path, relative index and metadata paths, metadata recomputation, repeated reads without mutation, and successful downstream workflow consumption from the persisted dataset directory. The test uses no mocks, downloads, or containers. Validation: the regression fails against the pre-fix code at the metadata/index mismatch. Both chaining cases and the existing populate-mode integration test pass. Focused bundle/import, dynamic-options, datatype, job-wrapper, finishing-state, and expression-tool tests: 104 passed, 1 skipped. tox -e lint,format,mypy passes, including mypy over 2392 source files. These are local results; the change has not been deployed or verified on the live service.
30 lines
1.1 KiB
XML
30 lines
1.1 KiB
XML
<tool id="data_manager_bundle_chain" name="Bundle chaining fixture" tool_type="manage_data" version="1.0" profile="24.0">
|
|
<command detect_errors="exit_code"><![CDATA[
|
|
python '$__tool_directory__/data_manager_bundle_chain.py' '$out_file'
|
|
#if $source.kind == 'bundle'
|
|
--database '$source.database.fields.path'
|
|
#end if
|
|
]]></command>
|
|
<inputs>
|
|
<param name="shape" type="select">
|
|
<option value="dict">Bare dict without dbkey</option>
|
|
<option value="list">List with dbkey</option>
|
|
</param>
|
|
<conditional name="source">
|
|
<param name="kind" type="select">
|
|
<option value="new">Create database</option>
|
|
<option value="bundle">Read database from bundle</option>
|
|
</param>
|
|
<when value="new" />
|
|
<when value="bundle">
|
|
<param name="database" type="select">
|
|
<options from_data_table="testbeta" />
|
|
</param>
|
|
</when>
|
|
</conditional>
|
|
</inputs>
|
|
<outputs>
|
|
<data name="out_file" format="data_manager_json" />
|
|
</outputs>
|
|
</tool>
|