Boot runs a full populate, so raising here stopped Galaxy from
starting where the eager toolbox, which never lists the store, starts
fine. Log the skipped confs and continue, and test that a full populate
completes after a listing failure.
A full populate replaces the default index with this run's
discoveries, so skipping the confs routed to that store dropped their
installed tools from the panel. Raise as the datatypes registry failure
does; a partial populate still only warns.
The test kept each conf's installed tools without the per-store
listing too. It now asserts one list_source_paths call for the two
confs routed to the read-only store.
Every conf routed to a read-only store ran store.list_source_paths(),
a full SELECT over the store, so several confs sharing one published
bundle repeated it, and each install's partial populates paid it too.
Cache the listing per store. A store whose listing fails is skipped for
every conf routed to it, with one warning.
indexed_at was taken after the parse and store phase, later than the
md5 recorded on the entry. A tool file edited in between got the old
hash and a newer indexed_at, so the reload re-index check skipped it.
The entry now carries the time taken just before hashing.
A shed update indexes the new revision ad hoc, as a storage-only entry,
before its conf entry catches up. The configured-entry guard kept the old
revision's entry, whose directory the update may already have moved, so
GET /api/tools failed to materialize the tool until the conf-driven
populate ran (TestCachedRepositoryInstallIntoReadOnlyStoreConf::
test_repository_update). Partial populates and runtime index writes now
supersede the slot; only a full scan's own storage-only twins still defer
to the configured entry.
Fixes#24263.
- A storage-only (unconfigured) version no longer becomes a tool's default
or a version offered by lookups, lineages or the registries. Indexes
loaded from a store, including published bundles, are repaired at load
time so the index schema hash stays the same.
- A toolbox reload re-indexes tool files in writable stores that changed
since they were indexed. A partial populate drops versions an edited file
no longer defines while keeping the tool's panel placement, and peer
processes drop those versions from their registries and lineages.
- The cached toolbox refuses to build while a read-only tool source store
has no compatible index, instead of populating each of its tools one at a
time. Startup fails with a ConfigurationError; a reload keeps the current
toolbox.
A repository update rewrites its tool files in place under the same guids,
sometimes only in the macro files they import. index_tool_files now parses
every requested file (incremental=False), because the populator's
unchanged-file shortcut hashes only the tool XML and misses macro changes.
A partial populate whose entries and placements already match the index
skips the index write and the reload broadcast, so a metadata reset or
update of an indexed repository costs one parse per tool. index_tool_files
reloads only after a write.
DataManagers registers a manager under DataManager.id, which is the
conf guid when there is one (every shed-installed data manager) and the
id attribute otherwise. Discovery recorded the id attribute, and the
load-time stamp kept whatever id an entry already had, so after a
populate a materialised DataManagerTool looked up "repeatmasker_download"
in a registry keyed by the guid, and every data manager job failed in
exec_after_process with "Invalid data manager ... requested".
Discovery now records guid-or-id, and the load-time stamp replaces a
differing id, which also corrects entries already in published stores.
A shed conf routed to a read-only store (a published CVMFS bundle) gains
tools on every install, but populates only walked confs routed to
writable stores. The installed tool's index entry was the ad-hoc one:
not in the panel, no section, no repository metadata, no placement. The
cached toolbox builds its panel from index placements, so the tool left
the panel and search at the next toolbox reload, in the installing
process, and never reached the panel of any other process. A full
populate dropped the entry altogether, after which the tool returned
404.
Walk confs routed to a read-only store for the tools that store does
not hold (only the requested paths in a partial populate), and index
them in the default store with their conf context. Confs whose store is
unavailable are already walked eagerly by the toolbox and are skipped.
populate_store_inline(paths=[...]) ran full discovery over every
writable conf (an existence check per tool, tool_dir walks, converter
and data manager checks) and only then restricted the result to the
requested paths. On test.galaxyproject.org each single-file populate
discovered 2466 tools in 6-9 s.
discover_tools() takes only_paths and skips the existence checks and
tool_dir walks for anything else. Conf context (section, guid, shed
coordinates, store routing) still comes from the conf walk.
Object identity shows the read-only store reuses its decoded index, and
discovery runs against a real GalaxyAppConfiguration. The populator
parse-count test only pinned an implementation detail.
Metadata generation loads each tool of a newly cloned repository before
its conf entry exists, so with the cached toolbox every load missed the
index and ran its own populate, reload_tool_source_cache broadcast and
index reload. A 30-tool repository paid that 30 times (16 s against 9 s
eager on a 9300-tool store). Index all of the repository's tool files in
one populate before loading them.
Every shed data manager entry names the shed conf it was installed into,
and discovery parsed that whole conf per data manager to read its
tool_path. With Main's shed conf (~9000 tools) and ~67 data managers this
was ~17 s per populate, paid twice per shed install with the cached
toolbox, even when the shed conf routes to a read-only store and is
otherwise skipped.
Hold the application toolbox lock across discovery and the final index write for in-process population. This prevents an uninstall from pruning an entry between a stale scan and its commit, only for the populator to restore that entry afterward.
Bundled tools can be visible through both the legacy tools/ tree and lib/galaxy/tools/bundled. The source-store discovery pass keyed deduplication by path, so these hard-linked aliases produced two placements for one tool: the configured section placement and a second top-level bundled placement. Unconfigured bundled and internal tools were also treated as panel entries even though eager mode only keeps them available for direct loading.
Recognize the compatibility aliases, mark bundled, hidden-lib, converter, data-manager, and ad-hoc discoveries as storage-only, and honor in_panel during runtime stub registration. Bump the index semantic revision so existing writable stores rebuild instead of retaining bad placements, and remove unresolved tool placeholders so an integrated panel already written by the buggy path repairs itself on startup.
Keep CachedTool proxies stable in toolbox registries and make the size-limited LRU the only registry that retains parsed Tool instances. This makes eviction actually release heavyweight tools instead of leaving them reachable through proxy and lookup maps.
Serialize concurrent cache misses per tool, classify materialization by reason, default unknown attribute access to strict failure, and retain an opt-in permissive compatibility setting. Stored-source and override failures now preserve their causes through ToolMaterializationError.
Serve requirements, containers, runtime-environment policy, and real-job action policy from indexed metadata so dependency summary and container-resolution scans do not parse the toolbox. Materialize explicitly at dependency mutation and job execution boundaries.
Cover LRU eviction, single-flight parsing, stable refresh identity, indexed dependency metadata, version lineage, strict/permissive behavior, and job-wrapper promotion.
The winner among same-id/same-version twins is decided by index add-order,
and two things made that order unstable run to run: pool results were
consumed as_completed (thread timing), and carried-forward entries were
added before re-parsed ones — so the run-N winner came back via the fast
path while its twin re-parsed and won run N+1. The flipping winner changed
its panel section in the whoosh corpus, so the corpus signature never
matched and the rebuild skip never fired (observed: corpus.md5 changing
across back-to-back no-change populates).
Collect one per-store input stream — carried entry or parsed triple — in
discovery (conf document) order and build the index from it. Twins now
resolve to the last conf entry, matching the eager toolbox, on every run.
The regression test fails on the previous code at parallel=4 and passes
now across three consecutive populates.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
A full populate re-parsed (macro-expanded) every tool and rebuilt the whole
index each run, so a re-run was never cheap even though nothing changed. Add
an incremental fast path: in the parallel workers, before parsing, compare
the tool file's raw md5 (md5_hash_file — the same signal the eager ToolCache
uses for reload detection) against the hash recorded on the tool's prior
index entry. If it matches, carry the entry forward and skip the parse
entirely. Requiring an existing entry means an empty or missing index can
never blank the store; prune forces a from-scratch re-parse.
The hash lives on the ToolIndexEntry (source_path + file_hash, both already
on the lazy branch), which is rebuilt from every run's tools, so it persists
whether a tool was freshly stored or content-deduped — no per-tool store
round-trip in the check. Pre-existing stores upgrade on the first run.
Locally a second populate reports the bulk of tools as "unchanged" (parse
skipped) and stays there across further runs; the remainder are twins (same
id+version at multiple paths share one entry) that still re-parse.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
- TestBroadcastReload (lazy): verified which collaborator a two-line
if/else router called, patching both branches — interaction assertions
with no observable effect.
- test_returns_false_on_connection_error: deleted kombu from sys.modules
(a no-op, names already bound) and only asserted isinstance(result, bool).
- three TestComputeHash cases that re-derived hashlib.sha256(...) to check
compute_hash equals it; the format/determinism/difference cases stay.
Frees the hashlib / pytest / populator_module imports.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Mirror the existing notify_callable seam: ToolFileWatcher now takes a
populate_callable (default populate_store_inline) and calls self._populate.
The watcher tests inject a recording fake and assert on the paths it
received, replacing three monkeypatch.setattr(populator, "populate_store_inline")
patches with dependency injection. Store-infra also gains the macro-repopulate
test (previously lazy-only).
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
- ToolFileWatcher now delegates filesystem observation and change detection
to galaxy.util.watcher.Watcher (the plumbing the eager toolbox watchers
build on) instead of hand-rolling an Observer + debounce Timer. The
per-file md5 re-check replaces the batch debounce, so --debounce is gone.
- send_reload_notification was broken on the kombu sqlalchemy transport: it
published with no declare list, so with in-memory routing the message
reached no consumer. It now declares the active per-process control queues,
read from the main DB (config.database_connection) via the new
galaxy.queues.control_queues_for_session helper split out of
all_control_queues_for_declare. In-process callers already route through
send_control_task; this fixes the app-less CLI path.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Every field parse was wrapped in its own try/except that swallowed to a
default, all inside the function-level try/except that already drops and
logs a bad entry. Call the parsers directly and let the outer handler own
failures — a tool whose source cannot be parsed (including a newer-profile
tool with no version) is dropped and logged rather than half-built from
defaults. The stub-based unit test gains parse_profile, which the old
version try/except had been masking as missing.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Two review fixes to the populator:
- Macro watch: the watcher filtered "macro" out of watched paths by
filename, so editing a macros file left every tool that <import>s it with
stale expanded content in the store. Accept every .xml and, on a <macros>
root, re-populate the tool siblings in the macro's directory.
- Typing: annotate build_index_entry_from_source (ToolSource /
StoredToolSource / DiscoveredTool) and return, and drop the hasattr
guards the ToolSource ABC already guarantees (parse_uuid stays guarded —
it is XML-only). Route the stub-based test through cast helpers now that
the function is typed.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Local imports are reserved for genuine circular-import problems; none
of these were. The watchdog optional dependency moves to the guarded
module-level try/except pattern galaxy.util.watcher uses, with --watch
raising when it isn't installed.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, and a sa_session parameter threaded through the factory,
populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone. The ToolSourceStore
ABC loses commit() (every backend persists per write). The dev/watch-mode
watcher now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
is_local defaulted to True, which misreported shed-installed tools (which
carry repository metadata) as local. Derive it from the discovered guid:
shed tools have a guid, local tools do not.
Synced from lazy-toolbox-atomic.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The index format promises guid-keyed shed entries, but
build_index_entry_from_source keyed every entry by the XML body's short
id and never populated tool_shed/repository_name/repository_owner/
changeset_revision. Installed tools ended up findable only by source
path, as short-id stubs with no repository metadata.
Discovery now reads the shed conf <tool> child elements (the same ones
the eager walk consumes via get_tool_repository_from_xml_item), and the
populator prefers the conf guid for both the store row's tool_id and
the index entry id.
Synced from lazy-toolbox-atomic.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).
- Backends: database (store-owned tool_source_record + tool_index tables,
migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
CVMFS-shipped), and a composite that layers named stores over the
default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
sources + index + whoosh search index; --watch updates the store on
file changes and broadcasts reload_tool_source_cache. The populator is
the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
whose rows belong to the job-request path and carry a raw-string
payload contract.
Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.