760 Commits
Author SHA1 Message Date
github-actions[bot] 6bb1ef06c1 chore(master): release 2.8.0 (#542)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.8.0
2026-10-10 10:52:52 +00:00
vastsa 63e72c1455 feat: add MCP quick share and safe HTML preview 2026-10-10 18:52:28 +08:00
github-actions[bot] 648d578d53 chore(master): release 2.7.1 (#536)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.7.1
2026-09-19 18:06:24 +00:00
vastsa 031d300cae fix: keep Docker as root, close s3_hostname SSRF, unbreak 404 tests
Revert gosu/non-root entrypoint so NAS and root-owned volumes keep
working. Reject @/userinfo in s3_hostname. Theme-less checkouts no
longer crash the HTML 404 handler.
2026-09-20 02:05:57 +08:00
Marrrrrrrrry 57c75a7141 全量修复/补测试+UED改进 (#535)
* fix: constant-time comparison for download tokens (security item 5)

Replace the set-membership key check in /share/download with
hmac.compare_digest over the two valid window tokens; add negative-path
tests (wrong key 403, both windows accepted, foreign-code token 403).

* fix: single source for attachment headers + guard tests (security item 2)

All six Content-Disposition construction sites across the five storage
backends now go through build_attachment_headers; the header is what
neutralizes stored XSS on the same-origin download path. Guard tests
assert every get_file_response uses the builder and no hand-built
disposition reappears.

* fix: reject internal-network endpoints for storage config (security item 7)

s3_endpoint_url/s3_hostname/webdav_url are fetched server-side; with
APP_ENV=production the write entry now enforces http(s) schemes and a
loopback/private/link-local/metadata host blacklist. Development env
stays open for local minio/webdav. Already-stored values are never
re-validated, so existing deployments are unaffected.

* fix: JSON 404 for handler-raised 404s; negative-path test batch

Starlette status-code handlers take precedence over the HTTPException
class handler, so registering the theme index as the 404 handler turned
every HTTPException(404) app-wide into a 200 HTML page (found by the new
negative-path tests). The new not_found_handler serves the theme page
only to browsers (Accept: text/html) and JSON 404 to API clients.

Negative-path batch: download count exhaustion, missing chunk session
404s, expired presign session deletion, admin update_file uniqueness/
existence.

* fix: run container as non-root app user with compat volume chown (security item 3)

Entrypoint starts as root only to fix the data-dir ownership (skipped
when nothing needs changing or when the deployer sets an explicit user),
then drops to uid 10001 via gosu. Verified in-container: uvicorn runs as
app (uid 10001), setup and share round-trip work against a root-owned
volume.

* fix: compare download tokens as bytes (non-ASCII key caused 500)

* test: lock changed-only semantics for endpoint config saves

Integration regression for the background lesson: a stored internal
endpoint must not block unrelated settings saves in production mode;
changing to an internal endpoint (URL or bare-hostname form) is 400.

* fix: resolve-based SSRF check — deny IP shorthands, hex/decimal IPs, DNS maps

Probe found the static blacklist bypassed by glibc shorthands (127.1,
10.1), decimal (2130706433) and hex IP forms, *.localhost, and
loopback-mapping DNS services. Both validators now resolve the host and
re-check every resolved address; hostname tier extracts the host part
correctly for [v6]/host:port/bare-v6 forms. 44 tests.

* test: pin 404 handler dual-branch behavior (browser theme page vs API JSON)

* fix: write back normalized endpoint values; hostname dev-gate test

* test: header-injection negative paths for attachment disposition

* fix: S3 merge buffers parts to >=5MiB; missing object 404 upfront

- S3 multipart rejects parts <5MiB (EntityTooSmall) except the last; chunk
  size is client-controlled (commonly 2-4MB) so multi-chunk merges always
  failed with 500. Merge now buffers chunks into >=5MiB parts (memory
  bound: 5MB + one chunk).
- get_file_response: a 404 head_object now raises StorageError(404)
  upfront instead of signing a doomed presigned URL and failing mid-stream
  (aligns with local backend semantics from the M2 behavior unification).

* test: S3 backend coverage via in-process moto server (13 cases)

mock_aws cannot intercept aioboto3's aiohttp stack, so tests run against
a real in-process ThreadedMotoServer endpoint. Covers roundtrips, missing
object 404, merge failure modes (missing chunk / hash mismatch -> abort,
no leftover object), cleanup scoping, presign shapes, proxy dispatch.
Also adds moto[s3,server] to the dev group and CI.

* fix: WebDAV missing object 404 upfront; connection errors map to 503

HEAD 404 was silently swallowed (signed a 200 bad stream that failed
mid-download, same family as the S3 bug), and HEAD connection errors were
also swallowed so the 503 mapping never fired. Both now surface properly;
the aiohttp session is reclaimed on pre-stream exception paths.

* test: WebDAV backend coverage via in-process WebDAV server (9 cases)

Real HTTP (HEAD/GET/PUT/DELETE/MKCOL/PROPFIND) against a minimal aiohttp
WebDAV server over a temp dir: roundtrips, missing object 404, connection
error 503, delete with empty-parent cleanup, merge failure modes, chunk
cleanup scoping.

* fix: OneDrive missing object maps to 404 upfront; OpenDAL behavior pinned

OneDrive get_file_response translated graph itemNotFound into the outer
503 catch-all; now raises StorageError(404) like local/S3/WebDAV.
OpenDAL already mapped missing objects to 404 via its outer catch —
pinned with fake-operator tests (SDK not in runtime deps, instances built
via __new__). Also tightens the S3 presign URL assertion.

* test: admin write-path coverage (batch/single delete, batch update, policy actions)

13 negative-path cases for the previously untested data-modifying admin
endpoints: mixed-id aggregation with duplicates and missing records,
empty-list rejections, clear_expired_at permanence semantics, policy
action boundaries (zero limit 400, unknown action 400, missing 404),
and the DoesNotExist->404 mapping pinned for single delete.

* test: admin read-path and view-preset CRUD coverage (22 cases)

Detail/metadata/preview/admin-download/activities/local-lists+delete/
verify: missing-record 404s, note/tag truncation limits, preview max_chars
truncation, activity filtering and the 80-event clamp, local path
traversal rejection. Presets: update-vs-create by id, name truncation,
filter normalization clamps, 24-preset cap, delete-missing 404.

* refactor: move auth primitives to apps/base/auth, kill base->admin dep

JWT create/verify, bearer extraction and the share-upload gate move to
apps.base.auth (consumed by both surfaces); admin.dependencies keeps
admin session gating and service providers, re-exporting the primitives
for import-path compatibility. base.views no longer imports from
apps.admin — the last cross-app reverse dependency is gone.

* refactor: split admin services god-module (ConfigService/LocalFile out)

apps/admin/services.py (1697 lines) becomes FileService (~1200 lines) +
a compatibility facade re-exporting ConfigService, LocalFileService,
LocalFileClass and keyvalue_write_lock from their new homes
(config_service.py holds the D5 KeyValue lock; one-way imports, no
cycle). test_admin_security patch targets follow ConfigService to its
new module. Behavior unchanged; 247 tests green.

* fix: size-less uploads no longer crash (upstream seek bug)

validate_file_size's size-None branch called UploadFile.seek(0, 2),
which raises TypeError (UploadFile.seek takes one arg); the underlying
SpooledTemporaryFile.seek is sync, so awaiting it also fails. Use the
underlying file object with sync seek(0, SEEK_END). Surface discovered
by the mypy pilot run; regression test covers the size-None branch.

* chore: bump direct deps (patch/minor)

fastapi 0.139.2->0.141.1, pydantic 2.12.5->2.13.5, uvicorn 0.51.0->0.53.0,
aiohttp 3.14.2->3.14.3; lockfile regenerated. tortoise-orm 0.x->1.x major
deliberately deferred pending API-change review.

* ci: gradual mypy adoption via baseline ratchet

scripts/mypy_ratchet.py compares mypy output against
scripts/mypy-baseline.txt (line numbers stripped for edit stability):
new type errors fail CI, fixes are folded back via --regenerate.
Baseline starts at 30 known errors; the seek bug found in the pilot was
fixed before baselining. mypy joins the dev group and CI.

* fix: migrate WebDAV auth off aiohttp.BasicAuth (deprecated in 4.0)

Dependency-upgrade probe on aiohttp 3.14.3 surfaced the BasicAuth
deprecation; switch to aiohttp.encode_basic_auth() headers so the
aiohttp 4.0 upgrade does not break the WebDAV backend.

* chore: drop dead code (WebDAV _instance stub, unused LocalFileClass.write)

* test: file-validation negative paths (13 cases)

Magic-bytes spoofing (text-as-png, exe-as-pdf, mp4/webp box detection),
whitelist rule semantics (star/image-wildcard/content mismatch), chunk-0
header validation parity, and end-to-end UploadFile rejection.

* test: migration runner coverage (5 cases)

Full chain executes 001-007 in filename order and registers; rerun is
idempotent; pre-registered entries are skipped (DDL not re-executed);
the resulting schema matches deployment; a failing migration propagates
and is not registered. Uses a fresh :memory: DB without
generate_schemas so the migrations themselves build the schema (the
real deployment path).

* refactor: split core/storage.py into a package (per-backend modules)

core/storage.py (1400+ lines, six backends) becomes core/storage/ with
_base.py (data contracts, interface, shared header builder), and
local/s3/onedrive/opendal/webdav modules. The package __init__
re-exports the full historical public surface; test patch targets move
to the precise backend modules (core.storage.local.data_root etc.) and
the attachment guard scans the package. Import-only change; 264 tests
green. Storage-layer DI deferred: current tests construct instances
against patched settings, so the added surface isn't justified yet.

* refactor: TypedDict for IPRateLimit records

The rate-limit ledger mixed int and datetime under a loose Union typing,
producing 5 mypy noise errors. A TypedDict gives precise per-key types;
setdefault replaces the get-then-assign pattern. One mypy baseline error
family resolved; 264 tests green.

* chore: tighten mypy baseline after IPRateLimit TypedDict

* refactor: resolve all 26 baseline mypy errors; precise test assertions

Per-error treatment (no bulk script): honest annotations for
StoredDownload (Path/bytes/Callable), KeyValue.value (Any for JSONField),
dict[str, Any] for form/config dicts; signature split in create_token;
str() normalization in config int(); platform/type-ignore only where the
checker cannot follow (msvcrt, FastAPI Request injection). Baseline file
and ratchet compare now redundant at zero errors but kept as the guard
against regressions.

Tests: assertions tightened to status_code == 403 with HTTPException
raises; _create_share explicit parameters replace **extra passthrough;
seek-fix pointer-reset assertion persisted.
2026-09-20 03:01:08 +09:00
github-actions[bot] 82555c08ed chore(master): release 2.7.0 (#533)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.7.0
2026-09-16 10:31:45 +00:00
vastsa 93a30a257c feat: mint extract codes from data/local without re-upload
Admin can browse NAS files under data/local including subfolders and share
them in place. Shares use file_path=local-ref, skip storage quota, and never
delete the original file on expire.
2026-09-16 18:31:00 +08:00
github-actions[bot] 3c135eb29f chore(master): release 2.6.1 (#529)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.6.1
2026-09-14 13:53:56 +00:00
Lan 0d82fbe26a Merge pull request #528 from vastsa/fix/post-514-upgrade-regressions
fix: 存量 background 值不再阻塞后台设置保存(#514 升级回归)
2026-09-14 22:53:32 +09:00
vastsa eb2db8f507 fix: don't let a legacy background value block settings saves
#514 began validating `background` as an http(s) URL against the *merged*
config, so a value stored by an older release (a relative path, or one
containing a space/parenthesis — all legal before) made every subsequent
settings save return 400, including saves that never touched `background`.
Existing deployments could not change any setting at all.

Validate only values that actually change: an untouched legacy value is
kept as-is (it is still html-escaped on render), while any edit must pass
validation. Regression tests cover both directions and fail without this
change.

Also treat OverflowError from hashlib.scrypt as a non-matching hash so a
hand-crafted n/r/p cannot disturb the login path on runtimes that raise it.
2026-09-14 21:52:23 +08:00
github-actions[bot] 18d344cc9b chore(master): release 2.6.0 (#515)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.6.0
2026-09-14 13:47:00 +00:00
Lan 8215f91edf Merge pull request #514 from Marrrrrrrrry/fix/m0-stop-bleeding
工程化加固、安全与性能修复、API 契约统一 snake_case(含 CI 测试流水线)
2026-09-14 22:46:18 +09:00
Marrrrrrrrry 775f3fc4ce ci: add test pipeline and API contract guard
- ci.yml: ruff + pytest on push/PR, installing from the hashed lockfile
  with --require-hashes, plus a lockfile-vs-requirements consistency
  check (dependabot cannot regenerate the lockfile; drift was silent).
- test_api_contract.py: deep-scans public config, dashboard, admin file
  list and share metadata responses for camelCase keys - the regression
  net for the D7 public-config leak found during review.
2026-09-11 04:24:47 +08:00
Marrrrrrrrry 13f572a86a test: follow public-config snake_case rename in admin-address test 2026-09-11 04:11:47 +08:00
Marrrrrrrrry 800bac0128 fix: snake_case the public config contract (D7 follow-up)
build_public_config/build_public_meta served a mixed camel/snake
contract (allowedFileTypes, meta.features.*, api.legacyConfig,
limits.uploadWindowMinutes). Rename to snake_case; frontend reads
allowed_file_types only.
2026-09-11 04:09:25 +08:00
Marrrrrrrrry 8972344235 docs: rename legacy camelCase config keys to snake_case in guides
Sync zh/en guide docs with migration 007 key names (upload_size,
open_upload, enable_chunk, expire_style, error_count, etc.) so existing
deployments following the docs use the current schema.
2026-09-11 03:38:52 +08:00
Marrrrrrrrry 6e7580b8dc docs: refresh API reference - real version, snake_case fields, EN presign page
API version 2.1.0 -> 2.5.6; response examples use the single message
field; dashboard/list examples and params match current snake_case
contract; drop stale save_path from presign init response; add English
presign-upload page and sidebar entry; uploadSize -> upload_size.
2026-09-11 03:37:24 +08:00
Marrrrrrrrry c7b5eabe7c refactor: single canonical snake_case keys across admin/share APIs (D7)
Drop the message/msg envelope twin, all camel+snake dual-field response
pairs (file list/detail/policy/storage, dashboard counters, batch op
counts, status insights, presets, metadata), and camel request fields
(clearExpiredAt, downloadLimit, sortBy/sortOrder, maxChars). Stored
KeyValue JSON (metadata, presets, activities) keeps legacy-key read
fallbacks so existing deployments normalize transparently.
2026-09-11 03:10:04 +08:00
Marrrrrrrrry bb40f40a18 refactor: hoist all five upload-entry workflows into FileUploadService (2.3)
share_text/share_file/complete_upload/presign proxy/presign confirm now
delegate quota reservation, storage writes, record creation, rollback and
chunk cleanup to service methods; handlers keep auth, request validation,
rate limiting and response wrapping. validate_file_size and
PRESIGN_SESSION_EXPIRES move to services as shared single sources.
2026-09-11 02:56:13 +08:00
Marrrrrrrrry a49727b63d refactor: move admin update_file orchestration into FileService (2.3) 2026-09-11 02:56:02 +08:00
Marrrrrrrrry 0038636a2a fix: serialize KeyValue JSON read-modify-write sections behind a process lock (D5)
settings/activities/view-presets are whole-blob KeyValue rows; concurrent
admin operations could silently drop each other's writes (last-writer-wins).
Single-worker deployment assumption documented; lock is process-local.
2026-09-09 19:09:14 +08:00
Marrrrrrrrry 3c93718d4b fix: FileCodes.expired_count defaults to -1 (time-based) instead of instantly-expired 0 (D4) 2026-09-09 19:05:26 +08:00
Marrrrrrrrry baecebd81e perf: dashboard counters and storage quota use SQL aggregation (D3)
- quota.get_storage_usage (runs on every upload) now aggregates via Sum
  instead of loading every row
- dashboard simple counters (total/size/expired/active/text/chunked/used,
  yesterday/today, top suffixes, recent 8) moved to SQL; the per-row health
  rules engine keeps its single pass (documented)
2026-09-09 19:04:12 +08:00
Marrrrrrrrry bb95017aa3 fix: save_file accepts plain binary streams; repair admin local file share (D2)
- storage contract drops UploadFile in favor of BinaryIO (+content_type param),
  completing the framework decoupling of the storage layer
- LocalFileClass.read() leaked an open fd and its handle was passed where
  save_file expected UploadFile — /admin/local/share 500ed on every call;
  now reads bytes via with-block and shares correctly (regression tests added)
- OneDrive save_file additionally had save_path called as a function — fixed
2026-09-09 18:53:31 +08:00
Marrrrrrrrry 61d72ab6cc perf: cache settings refresh behind a 2s TTL, force refresh on write paths (D6) 2026-09-09 18:47:40 +08:00
Marrrrrrrrry ec424dd909 perf: make is_config_initialized cheap — no scrypt on the per-request probe (D1) 2026-09-09 18:33:27 +08:00
Marrrrrrrrry 5121db6edc feat: rename all config keys to snake_case with DB migration 007
- 20 camelCase keys (uploadSize, expireStyle, openUpload, ...) renamed across
  DEFAULT_CONFIG, settings accessors, setup form fields, and ConfigService
- migrations_007 renames keys inside the existing settings KeyValue row,
  preserving all values for existing deployments
- schema guard meta-test keeps future keys declared
2026-09-09 18:09:19 +08:00
Marrrrrrrrry c5d525df08 feat: declare missing config keys (opendal_scheme, chunk_expire_hours) and add config schema guard
- opendal_scheme was read directly by the OpenDAL backend but never declared
  in DEFAULT_CONFIG: selecting that backend crashed with AttributeError
- chunk_expire_hours existed only as getattr fallbacks in two modules
- refresh_settings now warns about unknown user_config keys
- meta-test scans the codebase so any future settings.X reference without a
  DEFAULT_CONFIG declaration fails the suite
2026-09-09 17:43:28 +08:00
Marrrrrrrrry 61724cbf32 chore: warn at startup when multi-worker deployment weakens in-process rate limiting 2026-09-09 17:37:06 +08:00
Marrrrrrrrry 6587ec86be refactor: extract setup wizard and public pages from main.py (945 -> 148 lines)
- apps/base/setup_wizard.py: form parsing rules, setup options, and the
  two admin-facing HTML pages
- apps/base/pages.py: setup/assets/index/robots/public-config routes and
  theme resolution (404 handler registered on the app in main.py)
- tests updated to import from the new modules
2026-09-09 17:25:42 +08:00
Marrrrrrrrry 832266101e refactor: storage returns StoredDownload data; view layer builds starlette responses 2026-09-09 17:13:34 +08:00
Marrrrrrrrry ad3d319362 refactor: storage raises framework-neutral StorageError rendered identically by app handler 2026-09-09 17:07:43 +08:00
Marrrrrrrrry 892fb24ffc refactor: storage layer accepts plain StoredFile refs and caller-fetched chunk records (core/ is now ORM-free) 2026-09-09 17:05:38 +08:00
Marrrrrrrrry 3396644da3 refactor: move config and tasks modules from core to apps/base (core no longer imports apps) 2026-09-09 17:00:42 +08:00
Marrrrrrrrry 81d33b70de refactor: extract shared chunk-record fetch and hash verification into storage base class 2026-09-09 16:41:29 +08:00
Marrrrrrrrry a75936d85a refactor: use rollback_saved_file helper in presign proxy/confirm workflows 2026-09-09 16:39:42 +08:00
Marrrrrrrrry 357ab2fb6f refactor: extract FileUploadService into apps/base/services.py with shared rollback helper 2026-09-09 16:38:37 +08:00
Marrrrrrrrry 443cfe7472 feat: upgrade password hashing to memory-hard scrypt with transparent rehash
- new hashes use scrypt (n=2^14, r=8, p=1); sha256 and plaintext formats
  keep verifying unchanged, so existing deployments are unaffected
- on successful login a legacy-stored admin password is rehashed to scrypt
  in the settings KeyValue row
- add password_needs_rehash() and extend scheme detection; stdlib only, no
  new dependency
2026-09-09 16:33:07 +08:00
Marrrrrrrrry bde1fee17e docs: document lockfile regeneration flow in dependabot config 2026-09-09 16:28:36 +08:00
Marrrrrrrrry 964cba3636 chore: install dependencies from hashed lockfile in Docker build 2026-09-09 16:28:14 +08:00
Marrrrrrrrry 4904695955 chore: add hashed universal lockfile (uv pip compile --generate-hashes) 2026-09-09 16:27:38 +08:00
Marrrrrrrrry 4cab625d5a fix: validate background config as http(s) URL to close CSS quote-breakout in 2023 theme 2026-09-09 16:21:35 +08:00
Marrrrrrrrry a8c4072497 test: cover presigned upload journey (proxy mode auto-confirm, status, cancel, size limit) 2026-09-09 16:17:41 +08:00
Marrrrrrrrry 2bc278a58e test: cover merge_chunks failure modes and temp-file hygiene 2026-09-09 13:51:36 +08:00
Marrrrrrrrry 3f58e42be1 test: cover the three data-deleting cleanup tasks (expired files, stale chunks, presign sessions) 2026-09-09 13:42:11 +08:00
Marrrrrrrrry 625e50ca95 test: add end-to-end ASGI journey tests for share, chunk, and auth flows 2026-09-09 13:37:33 +08:00
Marrrrrrrrry 177a212c0f test: add shared test helpers and ASGI fixtures, dedupe 10x boilerplate
- tests/helpers.py: SettingsOverrideMixin + init_memory_db()/close_db()
- tests/conftest.py: httpx ASGI client fixtures (db, client, initialized_client)
- enable pytest asyncio_mode=auto
- replace 6 duplicated SettingsOverrideMixin copies and 4 duplicated
  Tortoise.init blocks across the legacy unittest suite
2026-09-08 17:47:21 +08:00
Marrrrrrrrry d701c6e00e chore: engineering hardening - test tooling, lint, security fix, and cleanup
- add pyproject.toml with pytest config and dev dependency group (PEP 735)
- add ruff (core correctness rules) + pre-commit hook
- fix stored XSS: html.escape site config before index template injection,
  with regression tests
- log previously swallowed cleanup exceptions in upload rollback paths
- consolidate three duplicate path generators into build_file_path
  (also aligns chunk/presign date dirs to get_now() instead of local time)
- drop redundant ip_limit sync in load_config (refresh_settings covers all limiters)
- remove dead code (calculate_file_hash, commented-out block), two orphan
  guide docs, and stale .gitignore entries
- add dependabot for pip, docker, and github-actions
- bump starlette 1.3.1 -> 1.6.0 (no breaking API changes; verified with the
  full test suite plus an ASGI smoke test incl. Range downloads)
- skip theme asset tests when themes/ is not built locally
2026-09-07 23:43:50 +08:00
github-actions[bot] 6c76aabe1e chore(master): release 2.5.6 (#509)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v2.5.6
2026-09-01 16:08:44 +00:00
vastsa e08555a6e8 Merge fix/prod-log-noise-20260901 2026-09-01 23:56:25 +08:00