45 Commits
Author SHA1 Message Date
mvdbeek 2932c43f86 Keep a full populate going when a read-only store cannot list its paths
Boot runs a full populate, so raising here stopped Galaxy from
starting where the eager toolbox, which never lists the store, starts
fine. Log the skipped confs and continue, and test that a full populate
completes after a listing failure.
2026-10-28 19:23:29 +01:00
mvdbeek 637c409cc9 Fail a full populate when a read-only store cannot list its paths
A full populate replaces the default index with this run's
discoveries, so skipping the confs routed to that store dropped their
installed tools from the panel. Raise as the datatypes registry failure
does; a partial populate still only warns.
2026-10-28 19:19:29 +01:00
mvdbeek b3a2d80e21 Count read-only store listings in the shared-store populate test
The test kept each conf's installed tools without the per-store
listing too. It now asserts one list_source_paths call for the two
confs routed to the read-only store.
2026-10-28 19:18:30 +01:00
mvdbeek e187f4a211 List a read-only store's source paths once per populate
Every conf routed to a read-only store ran store.list_source_paths(),
a full SELECT over the store, so several confs sharing one published
bundle repeated it, and each install's partial populates paid it too.
Cache the listing per store. A store whose listing fails is skipped for
every conf routed to it, with one warning.
2026-10-27 14:06:24 +01:00
mvdbeek c85fedcfad Stamp index entries with the time their file was hashed
indexed_at was taken after the parse and store phase, later than the
md5 recorded on the entry. A tool file edited in between got the old
hash and a newer indexed_at, so the reload re-index check skipped it.
The entry now carries the time taken just before hashing.
2026-10-25 08:56:15 +01:00
John Chilton 743129e012 Test converter retention and registry failure handling for default-target populates 2026-10-25 08:52:01 +01:00
mvdbeek bd5ca46de1 Let a rescan replace a configured entry at the same tool version
A shed update indexes the new revision ad hoc, as a storage-only entry,
before its conf entry catches up. The configured-entry guard kept the old
revision's entry, whose directory the update may already have moved, so
GET /api/tools failed to materialize the tool until the conf-driven
populate ran (TestCachedRepositoryInstallIntoReadOnlyStoreConf::
test_repository_update). Partial populates and runtime index writes now
supersede the slot; only a full scan's own storage-only twins still defer
to the configured entry.
2026-10-25 01:03:09 +02:00
mvdbeek 6ba3ab9a35 Keep configured tool versions, re-index local edits and refuse unindexed read-only stores in the cached toolbox
Fixes #24263.

- A storage-only (unconfigured) version no longer becomes a tool's default
  or a version offered by lookups, lineages or the registries. Indexes
  loaded from a store, including published bundles, are repaired at load
  time so the index schema hash stays the same.
- A toolbox reload re-indexes tool files in writable stores that changed
  since they were indexed. A partial populate drops versions an edited file
  no longer defines while keeping the tool's panel placement, and peer
  processes drop those versions from their registries and lineages.
- The cached toolbox refuses to build while a read-only tool source store
  has no compatible index, instead of populating each of its tools one at a
  time. Startup fails with a ConfigurationError; a reload keeps the current
  toolbox.
2026-10-25 01:02:59 +02:00
mvdbeek b72522a478 Type the shared populate test config as GalaxyAppConfiguration
The populator entry points take a GalaxyAppConfiguration; the untyped helpers it replaced hid that from mypy.
2026-10-23 14:17:54 +02:00
John Chilton d4afc9148b Share the populate test config between toolbox and populate tests 2026-10-23 14:14:47 +02:00
mvdbeek d40962e14e Re-parse a repository's tools on update; write the index only on change
A repository update rewrites its tool files in place under the same guids,
sometimes only in the macro files they import. index_tool_files now parses
every requested file (incremental=False), because the populator's
unchanged-file shortcut hashes only the tool XML and misses macro changes.

A partial populate whose entries and placements already match the index
skips the index write and the reload broadcast, so a metadata reset or
update of an indexed repository costs one parse per tool. index_tool_files
reloads only after a write.
2026-10-19 11:32:45 +02:00
John ChiltonandClaude Opus 5.5 89a42af850 Import populator at the top of test_populate_store
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:32:45 +02:00
mvdbeek 2ef1a7e425 Index data manager tools under the data manager's registry id
DataManagers registers a manager under DataManager.id, which is the
conf guid when there is one (every shed-installed data manager) and the
id attribute otherwise. Discovery recorded the id attribute, and the
load-time stamp kept whatever id an entry already had, so after a
populate a materialised DataManagerTool looked up "repeatmasker_download"
in a registry keyed by the guid, and every data manager job failed in
exec_after_process with "Invalid data manager ... requested".

Discovery now records guid-or-id, and the load-time stamp replaces a
differing id, which also corrects entries already in published stores.
2026-10-19 11:32:33 +02:00
mvdbeek 6e3fc63f1d Index tools a read-only store does not hold with their conf context
A shed conf routed to a read-only store (a published CVMFS bundle) gains
tools on every install, but populates only walked confs routed to
writable stores. The installed tool's index entry was the ad-hoc one:
not in the panel, no section, no repository metadata, no placement. The
cached toolbox builds its panel from index placements, so the tool left
the panel and search at the next toolbox reload, in the installing
process, and never reached the panel of any other process. A full
populate dropped the entry altogether, after which the tool returned
404.

Walk confs routed to a read-only store for the tools that store does
not hold (only the requested paths in a partial populate), and index
them in the default store with their conf context. Confs whose store is
unavailable are already walked eagerly by the toolbox and are skipped.
2026-10-19 11:32:32 +02:00
mvdbeek 8cc698072c Discover only the requested paths in partial tool source populates
populate_store_inline(paths=[...]) ran full discovery over every
writable conf (an existence check per tool, tool_dir walks, converter
and data manager checks) and only then restricted the result to the
requested paths. On test.galaxyproject.org each single-file populate
discovered 2466 tools in 6-9 s.

discover_tools() takes only_paths and skips the existence checks and
tool_dir walks for anything else. Conf context (section, guid, shed
coordinates, store routing) still comes from the conf walk.
2026-10-19 11:32:32 +02:00
mvdbeek 861f328782 Drop monkeypatching from the new tool source store tests
Object identity shows the read-only store reuses its decoded index, and
discovery runs against a real GalaxyAppConfiguration. The populator
parse-count test only pinned an implementation detail.
2026-10-19 11:32:31 +02:00
mvdbeek 6e2ec0c877 Index a repository's tools in one populate during install
Metadata generation loads each tool of a newly cloned repository before
its conf entry exists, so with the cached toolbox every load missed the
index and ran its own populate, reload_tool_source_cache broadcast and
index reload. A 30-tool repository paid that 30 times (16 s against 9 s
eager on a 9300-tool store). Index all of the repository's tool files in
one populate before loading them.
2026-10-19 11:32:31 +02:00
mvdbeek 48c3413416 Derive the populator's store set from conf_to_store_map
_build_stores and conf_to_store_map both parsed every tool conf for its
store name, so each populate parsed a large shed conf one extra time.
2026-10-19 11:32:31 +02:00
mvdbeek 38a7801655 Parse each shed conf once when discovering data manager tools
Every shed data manager entry names the shed conf it was installed into,
and discovery parsed that whole conf per data manager to read its
tool_path. With Main's shed conf (~9000 tools) and ~67 data managers this
was ~17 s per populate, paid twice per shed install with the cached
toolbox, even when the shed conf routes to a read-only store and is
otherwise skipped.
2026-10-19 11:32:30 +02:00
mvdbeek 07e22ed37c Serialize inline tool-store population with toolbox mutations
Hold the application toolbox lock across discovery and the final index write for in-process population. This prevents an uninstall from pruning an entry between a stale scan and its commit, only for the populator to restore that entry afterward.
2026-07-28 17:29:10 +02:00
mvdbeek fc7e9ebaf9 Address cached toolbox review feedback 2026-07-28 17:29:08 +02:00
mvdbeek ee4fcbbfdc Deduplicate tool store populator tests 2026-07-28 17:29:08 +02:00
mvdbeek 849266dc06 Fix cached shed tools missing from panel 2026-07-28 17:29:08 +02:00
mvdbeek 9c5ff40f5a Fix tool source store lint and typing 2026-07-28 17:29:08 +02:00
mvdbeek 8c605dca4e Add versioned external tool source stores 2026-07-28 17:29:08 +02:00
mvdbeek f6767ec95e cached toolbox: exclude storage-only tools from panel
Bundled tools can be visible through both the legacy tools/ tree and lib/galaxy/tools/bundled. The source-store discovery pass keyed deduplication by path, so these hard-linked aliases produced two placements for one tool: the configured section placement and a second top-level bundled placement. Unconfigured bundled and internal tools were also treated as panel entries even though eager mode only keeps them available for direct loading.

Recognize the compatibility aliases, mark bundled, hidden-lib, converter, data-manager, and ad-hoc discoveries as storage-only, and honor in_panel during runtime stub registration. Bump the index semantic revision so existing writable stores rebuild instead of retaining bad placements, and remove unresolved tool placeholders so an integrated panel already written by the buggy path repairs itself on startup.
2026-07-28 17:29:07 +02:00
mvdbeek e687a53eab Make cached tool materialization bounded and explicit
Keep CachedTool proxies stable in toolbox registries and make the size-limited LRU the only registry that retains parsed Tool instances. This makes eviction actually release heavyweight tools instead of leaving them reachable through proxy and lookup maps.

Serialize concurrent cache misses per tool, classify materialization by reason, default unknown attribute access to strict failure, and retain an opt-in permissive compatibility setting. Stored-source and override failures now preserve their causes through ToolMaterializationError.

Serve requirements, containers, runtime-environment policy, and real-job action policy from indexed metadata so dependency summary and container-resolution scans do not parse the toolbox. Materialize explicitly at dependency mutation and job execution boundaries.

Cover LRU eviction, single-flight parsing, stable refresh identity, indexed dependency metadata, version lineage, strict/permissive behavior, and job-wrapper promotion.
2026-07-28 17:28:06 +02:00
mvdbeek 5ed2273b08 source store: satisfy format and type checks 2026-07-13 11:27:58 +02:00
mvdbeek fc7aa2cd41 source store: correct storage documentation 2026-07-13 11:22:58 +02:00
mvdbeek acb7617c22 source store: index tests and dependency metadata 2026-07-13 11:22:45 +02:00
mvdbeek b1d9bba085 source store: carry lazy index contract 2026-07-13 10:59:40 +02:00
mvdbeek 2604501df1 Add missing type hints for tool source store changes 2026-07-13 10:27:05 +02:00
mvdbeek 3b10e156c7 source_store: build the index from discovery-ordered inputs
The winner among same-id/same-version twins is decided by index add-order,
and two things made that order unstable run to run: pool results were
consumed as_completed (thread timing), and carried-forward entries were
added before re-parsed ones — so the run-N winner came back via the fast
path while its twin re-parsed and won run N+1. The flipping winner changed
its panel section in the whoosh corpus, so the corpus signature never
matched and the rebuild skip never fired (observed: corpus.md5 changing
across back-to-back no-change populates).

Collect one per-store input stream — carried entry or parsed triple — in
discovery (conf document) order and build the index from it. Twins now
resolve to the last conf entry, matching the eager toolbox, on every run.
The regression test fails on the previous code at parallel=4 and passes
now across three consecutive populates.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:05 +02:00
mvdbeek d6422b6012 source_store: skip re-parsing unchanged tools on incremental populate
A full populate re-parsed (macro-expanded) every tool and rebuilt the whole
index each run, so a re-run was never cheap even though nothing changed. Add
an incremental fast path: in the parallel workers, before parsing, compare
the tool file's raw md5 (md5_hash_file — the same signal the eager ToolCache
uses for reload detection) against the hash recorded on the tool's prior
index entry. If it matches, carry the entry forward and skip the parse
entirely. Requiring an existing entry means an empty or missing index can
never blank the store; prune forces a from-scratch re-parse.

The hash lives on the ToolIndexEntry (source_path + file_hash, both already
on the lazy branch), which is rebuilt from every run's tools, so it persists
whether a tool was freshly stored or content-deduped — no per-tool store
round-trip in the check. Pre-existing stores upgrade on the first run.

Locally a second populate reports the bulk of tools as "unchanged" (parse
skipped) and stays there across further runs; the remainder are twins (same
id+version at multiple paths share one entry) that still re-parse.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:04 +02:00
mvdbeek 67e7ee6e65 test/populate_store: drop low-value tests
- TestBroadcastReload (lazy): verified which collaborator a two-line
  if/else router called, patching both branches — interaction assertions
  with no observable effect.
- test_returns_false_on_connection_error: deleted kombu from sys.modules
  (a no-op, names already bound) and only asserted isinstance(result, bool).
- three TestComputeHash cases that re-derived hashlib.sha256(...) to check
  compute_hash equals it; the format/determinism/difference cases stay.

Frees the hashlib / pytest / populator_module imports.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:04 +02:00
mvdbeek 9b65da1161 ToolFileWatcher: inject the populate callable instead of patching a global
Mirror the existing notify_callable seam: ToolFileWatcher now takes a
populate_callable (default populate_store_inline) and calls self._populate.
The watcher tests inject a recording fake and assert on the paths it
received, replacing three monkeypatch.setattr(populator, "populate_store_inline")
patches with dependency injection. Store-infra also gains the macro-repopulate
test (previously lazy-only).

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:03 +02:00
mvdbeek 0ce590434b source_store watch mode: reuse galaxy.util.watcher; fix CLI reload broadcast
- ToolFileWatcher now delegates filesystem observation and change detection
  to galaxy.util.watcher.Watcher (the plumbing the eager toolbox watchers
  build on) instead of hand-rolling an Observer + debounce Timer. The
  per-file md5 re-check replaces the batch debounce, so --debounce is gone.

- send_reload_notification was broken on the kombu sqlalchemy transport: it
  published with no declare list, so with in-memory routing the message
  reached no consumer. It now declares the active per-process control queues,
  read from the main DB (config.database_connection) via the new
  galaxy.queues.control_queues_for_session helper split out of
  all_control_queues_for_declare. In-process callers already route through
  send_control_task; this fixes the app-less CLI path.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:03 +02:00
mvdbeek ab950af67f source_store populator: drop per-parse try/except in build_index_entry_from_source
Every field parse was wrapped in its own try/except that swallowed to a
default, all inside the function-level try/except that already drops and
logs a bad entry. Call the parsers directly and let the outer handler own
failures — a tool whose source cannot be parsed (including a newer-profile
tool with no version) is dropped and logged rather than half-built from
defaults. The stub-based unit test gains parse_profile, which the old
version try/except had been masking as missing.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:03 +02:00
mvdbeek ba6f41b317 source_store populator: macro-file watch + typed build_index_entry_from_source
Two review fixes to the populator:

- Macro watch: the watcher filtered "macro" out of watched paths by
  filename, so editing a macros file left every tool that <import>s it with
  stale expanded content in the store. Accept every .xml and, on a <macros>
  root, re-populate the tool siblings in the macro's directory.

- Typing: annotate build_index_entry_from_source (ToolSource /
  StoredToolSource / DiscoveredTool) and return, and drop the hasattr
  guards the ToolSource ABC already guarantees (parse_uuid stays guarded —
  it is XML-only). Route the stub-based test through cast helpers now that
  the function is typed.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:02 +02:00
mvdbeek f40e273d6b source_store: hoist local imports to module level
Local imports are reserved for genuine circular-import problems; none
of these were. The watchdog optional dependency moves to the guarded
module-level try/except pattern galaxy.util.watcher uses, with --watch
raising when it isn't installed.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:27:02 +02:00
mvdbeek 22c1f9ec5b Drop the main-DB tool source store backend
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, and a sa_session parameter threaded through the factory,
populator, watcher and CLI.

Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone. The ToolSourceStore
ABC loses commit() (every backend persists per write). The dev/watch-mode
watcher now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:59 +02:00
mvdbeek 761e43d544 Relocate tool_source_store to galaxy.tools.source_store
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:59 +02:00
mvdbeek 24972106e4 populator: stamp ToolIndexEntry.is_local from the discovered guid
is_local defaulted to True, which misreported shed-installed tools (which
carry repository metadata) as local. Derive it from the discovered guid:
shed tools have a guid, local tools do not.

Synced from lazy-toolbox-atomic.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:58 +02:00
mvdbeek b571e0307a populator: key shed entries by guid, stamp repository metadata
The index format promises guid-keyed shed entries, but
build_index_entry_from_source keyed every entry by the XML body's short
id and never populated tool_shed/repository_name/repository_owner/
changeset_revision. Installed tools ended up findable only by source
path, as short-id stubs with no repository metadata.

Discovery now reads the shed conf <tool> child elements (the same ones
the eager walk consumes via get_tool_repository_from_xml_item), and the
populator prefers the conf guid for both the store row's tool_id and
the index entry id.

Synced from lazy-toolbox-atomic.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:58 +02:00
mvdbeek 8162853207 Add pluggable tool source store infrastructure
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).

- Backends: database (store-owned tool_source_record + tool_index tables,
  migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
  CVMFS-shipped), and a composite that layers named stores over the
  default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
  sources + index + whoosh search index; --watch updates the store on
  file changes and broadcasts reload_tool_source_cache. The populator is
  the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
  against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
  whose rows belong to the job-request path and carry a raw-string
  payload contract.

Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
2026-07-13 10:26:57 +02:00