Address review comments on
https://github.com/galaxyproject/galaxy/pull/24907 :
- Move the release note about the removal of Python 3.10 support from
"Deprecation Notices" to "Admin Notes", as done for Python 3.8 in
`25.0_announce.rst`.
- Mention that the requirement also applies to the Python used on compute
nodes to run job-side code when Galaxy's virtualenv is not shared.
- Recommend Python 3.12 or 3.13, since Python 3.11 reaches its
end-of-life in October 2027, within the support period of Galaxy 26.2
(see
https://github.com/galaxyproject/galaxy/pull/24907#discussion_r4584251159
). TensorFlow, used for tool recommendations, doesn't support Python
3.14 yet.
- Raise the default `GALAXY_CONDA_PYTHON_VERSION` in
`scripts/common_startup.sh` from 3.11 to 3.13 accordingly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`tensorflow` 2.15.1 only supports Python up to 3.11, while 2.21.0 (the
latest release) supports Python 3.10 to 3.13. It also resolves against
Galaxy's pinned requirements apart from `h5py` (it needs `h5py<3.15.0`),
whereas 2.15.1 requires `numpy<2` and `wrapt<1.15`.
To keep `tensorflow` from downgrading the pinned `h5py` on every startup,
add `lib/galaxy/dependencies/conditional-overrides.txt`, which drops the
`h5py` upper bound, and pass it with `--override` to the conditional
dependency installation in `scripts/common_startup.sh` when it uses `uv`
(pip doesn't support overrides).
`tensorflow` 2.16 and later use Keras 3, which calls a layer's `call()`
without the `training` argument when building a functional model, so
`create_transformer_model()` failed with:
```
File "lib/galaxy/tools/recommendations.py", line 118, in create_transformer_model
x, weights = transformer_block(x)
^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/lib/python3.11/inspect.py", line 3195, in bind
return self._bind(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/inspect.py", line 3110, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'training'
```
Give `TransformerBlock.call()`'s `training` argument a default of `False`.
With this, loading the default tool recommendation model
(`tool_recommendation_model_v_0.2.hdf5`) with `tensorflow` 2.21.0 on
Python 3.11 and 3.13 gives the same predictions as with 2.15.1 (scores
within 6e-8).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python 3.10 reached its end-of-life in October 2026.
Also:
- Run `make pyupgrade` and `make format`
- Fix new ruff and mypy errors
- Run `uv lock` and `sh lib/galaxy/dependencies/update.sh --export-only`
- Run `make update-client-api-schema`
flake8-bugbear 26.10.27 now reports these expressions, whose result is
discarded, so they never did what they were meant to:
```
./lib/galaxy/datatypes/genetics.py:1153:21: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./lib/galaxy/datatypes/genetics.py:1155:21: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./lib/galaxy/datatypes/genetics.py:1160:21: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./lib/galaxy/tool_util/deps/brew_exts.py:379:9: B018 Found useless GeneratorExp expression. Consider either assigning it to a variable or removing it.
./lib/galaxy_test/api/test_workflows.py:950:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./lib/galaxy_test/api/test_workflows.py:961:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./lib/galaxy_test/api/test_workflows.py:969:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./scripts/cleanup_datasets/pgcleanup.py:1434:13: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
./test/unit/tool_util/verify/test_asserts.py:1096:5: B044 `assert <non-empty string>` is always true. Did you forget the condition?
```
`map()` and generator expressions are lazy, so replace them with `for`
loops. This means code that never ran before now runs:
- `DataIn.sniff()` now actually checks that the tokens of the first 3
lines are numeric.
- `EnvAction.build_env()` now actually applies the environment actions.
- `Cleanup` in `pgcleanup.py` now logs the documentation lines of each
action.
- The assertions in `TestWorkflowsApi.test_update` now run. The test
moved every step to position `(1, 1)`, but Galaxy normalises step
positions so that the minimum left and top are 0, so they came back
as `(0, 0)`. Use distinct, already normalised positions instead, and
check at the end that the original positions are restored, instead
of modifying the downloaded steps again.
The B044 assertion in `test_asserts.py` was missing its `in a`
condition; with it added, the expected message also needed the archive
member prefix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`pgcleanup.py` replaced `cleanup_datasets.py` in ba83310d53, but only the
dataset half of the legacy `delete_datasets()` was ported. The library half
(cleanup_datasets.py:407-440) is what marks every version of a deleted library
dataset as deleted before the library_dataset is purged, and `DeleteDatasets`
still documents that precondition ("all marked as deleted (LDDA) or purged
(HDA)").
Nothing else sets `ldda.deleted` today: the API only flags
`library_dataset.deleted` (services/library_contents.py:213,
api/library_datasets.py:362-368), so `DeleteDatasets` vetoes forever on any
dataset referenced by a library association and `PurgeDatasets` never sees the
row. Datasets that only live in libraries are therefore never cleaned up.
Add the missing `delete_library_datasets` action as a port of the legacy
behaviour, keeping legacy semantics deliberately: it marks the versions
deleted and purges the library dataset, so a purged library dataset can no
longer be undeleted (see the follow-up about enforcing that in the API).
`--force-retry` reconciles rows the legacy scripts left behind, i.e. library
datasets that are purged while their versions were never marked deleted.
`ldda.update_time` is intentionally not stamped so that
`delete_library_datasets delete_datasets purge_datasets` can be chained in a
single run; the retention decision is made on `library_dataset.update_time`,
as in the legacy script.
The folder/library halves of the legacy script (_purge_folder, purge_libraries)
are not included here.
Galaxy does not always follow a release with a minor increment (26.2 is followed by 27.0), so `make release-create-rc RELEASE_NEXT=27.0` passes the dev version to release.sh via the new -n option.
TypedDicts mirror the directive metadata interfaces in directives.ts;
passes mypy --strict.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Backend validator and client editor read one list; a SectionWrapper test
replaces the regex scrape of the .vue file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Move the tool state diagrams into a new doc/source/dev/tool_state.md
rendered by sphinxcontrib-mermaid, updating the state representations
to match galaxy.tool_util.parameters.state. Drop the PlantUML sources,
committed SVGs, jar-downloading image.Makefile and the dead
docs-slides targets and scripts/slideshow (doc/source/slideshow was
removed in 2019).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-fetches the schema set by walking the top-level schema's $refs and
verifies each response parses as JSON (the host's WAF can serve
challenge HTML with a 200 status). The README points to it so the
vendored files aren't edited by hand. Running it today reproduces the
vendored files byte for byte.
Conflict: packages/tool_util/setup.cfg was deleted on dev by the
pyproject.toml migration (c329529e51) and modified on release_26.1 by
#23256, which moved jsonschema from install_requires to the test extra.
Kept the deletion and ported that move into packages/tool_util/pyproject.toml
so the dependency change isn't lost in the merge forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The patch decrement only made sense for release-branch versions like
26.1.2.dev0. On dev the version is two-component (26.2.dev0), so the
decrement landed on the minor, and at 27.0.dev0 it produced 27.-1, which
pip rejects -- reintroducing the hard failure this is meant to remove.
Let the resolver pick the most recent published release below the current
version instead. Release candidates get the same treatment, since release
branches are cut straight to rc and sit there for weeks.
The bound uses the version with the .dev/rc suffix stripped: a bound that
is itself a prerelease would turn on prerelease matching and expose the
stray prerelease wheels on PyPI.
Planemo enables GALAXY_INSTALL_PREBUILT_CLIENT when serving Galaxy, so rejecting development versions breaks planemo serve against development branches.
Only tagged galaxy-web-client releases are published and they generally differ only slightly from the corresponding Galaxy development branch. Warn about that mismatch and install the preceding patch release for development versions, while continuing to pin tagged Galaxy versions exactly.
common_startup.sh resolved only galaxy.yml, so a Tool Shed deployment
installed nothing from conditional-requirements.txt. That is why the
sentry-sdk>=2.63.0 pin never reached toolshed.test, which kept an older
sentry-sdk and hit the FastAPI >= 0.137 wrapper-loop crash: sentry
re-wraps dependant.call on every request there, and inspect.unwrap
raises once the chain passes the recursion limit.
Move the machinery and the app-agnostic checks into
BaseConditionalDependencies so the Tool Shed evaluates the database
driver and sentry-sdk checks against its own config section, and let
run_tool_shed.sh select it with GALAXY_CONDITIONAL_DEPENDENCIES_APP.
Galaxy-only checks stay on ConditionalDependencies, where check() keeps
treating a missing check_<name> method as "not needed".
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).
- Backends: database (store-owned tool_source_record + tool_index tables,
migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
CVMFS-shipped), and a composite that layers named stores over the
default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
sources + index + whoosh search index; --watch updates the store on
file changes and broadcasts reload_tool_source_cache. The populator is
the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
whose rows belong to the job-request path and carry a raw-string
payload contract.
Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
Flip the relationship between directives.yml and the Python validator: the YAML
is now authoritative and scripts/markdown_directives_doc.py generates everything
downstream of it:
- lib/galaxy/managers/_markdown_directives.py - generated VALID_ARGUMENTS,
EMBED_CAPABLE_DIRECTIVES, SHARED_ARGUMENTS, DynamicArguments; markdown_parse
re-exports these, staying self-contained (no YAML at runtime) so it remains
reusable outside Galaxy (e.g. gxformat2).
- client .../Utilities/requirements.yml - generated from each directive's
'requires', removing the hand-maintained duplicate.
- directives.md - unchanged generator, now fed by the same source.
Add _shared_arguments to directives.yml (homes 'collapse' as data). Replace the
now-tautological yml-vs-parser cross-check with internal yml validation plus a
dispatch-coverage check against markdown_util.py (the gap that hid the dead
instance_citation_bibtex directive). Guard all three generated artifacts in
test_markdown_directives_doc.py and via make client-gen-markdown-directives.
Emit the directives.md tables pre-formatted to match prettier (column padding),
so the generated reference is prettier-stable and needs no ignore entry; drop the
ineffective client/.prettierignore line for it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enrich directives.yml into the source of truth for directive docs: per-param
type/context/default/description (via shared _parameter_sets), embeddable,
requires, category, renders; add missing instance_*_link and visualization
entries.
Add scripts/markdown_directives_doc.py to render directives.md and cross-check
the yml against markdown_parse.py (VALID_ARGUMENTS, EMBED_CAPABLE_DIRECTIVES,
SHARED_ARGUMENTS) + requirements.yml. Guard via test_markdown_directives_doc.py
and `make client-gen-markdown-directives`, mirroring config-rebuild. directives.md
is generated, so prettier-ignore it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop Python 3.8 support in 5 Pulsar-compatible packages
(job_metrics, tool_util, tool_util_models, util, objectstore)
and run pyupgrade --py310-plus on their source code.
- Bump requires-python from >=3.8 to >=3.10 in all 5 packages
- Remove Python 3.8 and 3.9 classifiers
- Remove ruff per-file-ignores for UP rules on these paths
- Remove backports.zoneinfo conditional dependency
- Remove pydyf<0.11 pin from conditional-requirements.txt
- Update Makefile pyupgrade target (remove PY38_PYUPGRADE_PATHS)
- Update CI workflow to test with Python 3.10 instead of 3.8
Clean up unused deprecated typing imports after pyupgrade
Remove now-unused typing imports (Dict, List, Optional, Set, Tuple,
Type, Union) that became dead after pyupgrade --py310-plus converted
annotations to use built-in types and | syntax.
Also run ruff check --fix --select=UP007,UP045 across the entire
codebase to convert remaining Optional[X] -> X | None and
Union[X, Y] -> X | Y patterns.
Enable ruff UP007/UP045 for Python 3.10 union syntax
Remove UP007 (Union[X,Y] -> X | Y) and UP045 (Optional[X] -> X | None)
from the ruff ignore list and convert all type aliases across the
codebase. These rules were deferred while Python 3.9 was supported;
requires-python is now >=3.10.
A custom script was used because neither ruff --fix nor
pyupgrade --py310-plus converts Optional[X]/Union[X,Y] in type alias
positions (e.g. X = Union[A, B]) — they only handle annotation
positions (e.g. def f(x: Optional[int])). All 167 violations were
module-level type aliases. A few edge cases were fixed manually:
single-element Union[X,], typing.Union qualified refs, runtime
Optional[type] calls, and Annotated[Optional[...]] pydantic fields.
Fix UP007 autofix regression with string forward reference type aliases
Commit fa6bd955a0 enabled ruff UP007/UP045 and auto-fixed
module-level type aliases using Union with string forward
references, producing invalid 'str | str' expressions.
This was a known ruff bug (charliermarsh/ruff#826) that has since
been fixed in later ruff versions, but this codebase was
converted before the fix was in place.
Revert to Union syntax and restore TYPE_CHECKING imports that
ruff's TCH rule cleaned up as a side effect when it thought the
forward references were unused.
Skips the physical file removal process for datasets stored in user-defined object stores. These storage locations cannot be resolved by the cleanup script, so the database records are updated while the external files remain intact to prevent potential errors or data loss.
- Extract FavoritesManager (lib/galaxy/managers/favorites.py): owns
normalize / resolve / persist / order logic; controllers in users.py
are now 2 lines each, and trans.sa_session.commit moves out of the
controller layer. Drops the redundant re-normalize on every write
and replaces FavoriteObjectType.X.value == .value defensive dispatch
with enum-identity comparison. Renames the set_favorite_order
endpoint summary away from the ambiguous "top-level" wording.
- Move curated_tool_tags_by_id off the toolbox into ToolsService;
the controller now routes through the service instead of poking
trans.app.toolbox directly. Raw aggregates (curated_tool_tags,
tool_edam_operations, tool_edam_topics) remain on the toolbox since
they're derived from per-tool data.
- Rename /api/tools/tags -> /api/tags/tool_tags (operation_id
tags__tool_tags) to disambiguate from /api/tools/{id}; update the
client store, tests, and the hand-edited OpenAPI schema entry.
(Run `make update-client-api-schema` before pushing to regenerate
the schema in its canonical alphabetical position.)
- Promote inline imports to module-level (ontology_data.py logging,
app/__init__.py configure_tool_tag_mapping); replace
getattr(self.config, "tool_tag_mappings_file", None) with direct
attribute access now that the option is registered in the schema.
- Convert ontology_data's import-time bundled-mapping reads to
lru_cache so the YAML / TSV files load lazily; fix the
Tool._setup_id comment to point at the actual lowercasing site
(Tool.parse around `self_ids = [self.id.lower()]`).
- doc/source/admin/tool_panel.rst: 26.0 -> 26.1.
- scripts/extract_tool_sections_from_api.py: annotate the public
helpers the test suite imports.
- test_to_dict_cache_drops_when_tool_removed now asserts observable
to_dict() output before/after remove, not private cache membership.
- Split test_tool_discovery_landing into focused tests; extract a
helper that opens the discovery view and types the standard filter.
- Move the Playwright drag helper from test_tool_panel_search.py into
TestWithSeleniumMixin.playwright_drag_item_above so future tests
reuse it.
- Client tests: drop the full-string Whoosh equality in
utilities.test.ts; hard-code expected Whoosh strings in
ToolsList.test.ts (drop createWhooshQuery as both sides of the
assertion) and remove the redundant injectTestRouter mount option
(ToolsList.vue only uses the useRouter composable); drop the
FontAwesome data-icon/iconName assertions in ToolPanel.test.ts in
favor of icon-existence checks; replace three near-identical
add/remove favorite tests in ToolsListCard.test.ts with an it.each;
export quoteToolTagValue from Panels/utilities so
filterConversion.test.js imports the real helper instead of
reconstructing it inline; delete the spy-only "loads the curated
tag mapping" test in ToolBoxSearch.test.ts (the gate is covered by
ToolPanel.test.ts's negative case and the favorite-tag rendering
tests cover the positive case end-to-end).
Two related fixes around the admin-facing helper script and admin docs:
* Make the helper script work from any cwd and against any Galaxy
server: `--api-url`, `--output`, `--timeout` argparse flags; raise
`RuntimeError` on transport / decode failures (was a silent no-op);
anchor the default output path on the script location so
`python scripts/extract_tool_sections_from_api.py` works without a
prior `cd`. Default output now points at `config/tool_tag_mappings.yml`
to match the new admin override path.
* Add a "Curated tool tags" section to `doc/source/admin/tool_panel.rst`
documenting the new `tool_tag_mappings_file` option, the new
`tag:"…"` / `ontology:"…"` search syntax, the `?section=` URL
deprecation, and the reindex requirement after changing the mapping.
Nest the prebuilt/hash-check branches inside `if [ $SKIP_CLIENT_BUILD
-eq 0 ]` so the shape mirrors the subsequent build-or-install block.
Per Nicola's review.
The prebuilt client path (GALAXY_INSTALL_PREBUILT_CLIENT=1, used by
planemo) now pulls the matching galaxy-web-client wheel from PyPI and
lets webapp.py pick it up via `import galaxy.web_client`. The
root-level pnpm install is no longer needed for this flow -- the
Galaxy client isn't a JS library, so shipping it as an npm package
was always the wrong shape.
Dev versions don't have a published wheel, so we bail early with a
message pointing users at the source build path.
No root pnpm-lock.yaml is shipped, so `pnpm install --frozen-lockfile`
fails with ERR_PNPM_NO_LOCKFILE (hit locally via planemo). The dev
branch is unaffected -- client/pnpm-lock.yaml exists.
only directories in `lib/galaxy` & `packages/*/galaxy`
All other files removed or moved to their own module
All references to those files updated, especially galaxy/version.py