Files
FileCodeBox/apps/base/services.py
T
Marrrrrrrrry 57c75a7141 全量修复/补测试+UED改进 (#535)
* fix: constant-time comparison for download tokens (security item 5)

Replace the set-membership key check in /share/download with
hmac.compare_digest over the two valid window tokens; add negative-path
tests (wrong key 403, both windows accepted, foreign-code token 403).

* fix: single source for attachment headers + guard tests (security item 2)

All six Content-Disposition construction sites across the five storage
backends now go through build_attachment_headers; the header is what
neutralizes stored XSS on the same-origin download path. Guard tests
assert every get_file_response uses the builder and no hand-built
disposition reappears.

* fix: reject internal-network endpoints for storage config (security item 7)

s3_endpoint_url/s3_hostname/webdav_url are fetched server-side; with
APP_ENV=production the write entry now enforces http(s) schemes and a
loopback/private/link-local/metadata host blacklist. Development env
stays open for local minio/webdav. Already-stored values are never
re-validated, so existing deployments are unaffected.

* fix: JSON 404 for handler-raised 404s; negative-path test batch

Starlette status-code handlers take precedence over the HTTPException
class handler, so registering the theme index as the 404 handler turned
every HTTPException(404) app-wide into a 200 HTML page (found by the new
negative-path tests). The new not_found_handler serves the theme page
only to browsers (Accept: text/html) and JSON 404 to API clients.

Negative-path batch: download count exhaustion, missing chunk session
404s, expired presign session deletion, admin update_file uniqueness/
existence.

* fix: run container as non-root app user with compat volume chown (security item 3)

Entrypoint starts as root only to fix the data-dir ownership (skipped
when nothing needs changing or when the deployer sets an explicit user),
then drops to uid 10001 via gosu. Verified in-container: uvicorn runs as
app (uid 10001), setup and share round-trip work against a root-owned
volume.

* fix: compare download tokens as bytes (non-ASCII key caused 500)

* test: lock changed-only semantics for endpoint config saves

Integration regression for the background lesson: a stored internal
endpoint must not block unrelated settings saves in production mode;
changing to an internal endpoint (URL or bare-hostname form) is 400.

* fix: resolve-based SSRF check — deny IP shorthands, hex/decimal IPs, DNS maps

Probe found the static blacklist bypassed by glibc shorthands (127.1,
10.1), decimal (2130706433) and hex IP forms, *.localhost, and
loopback-mapping DNS services. Both validators now resolve the host and
re-check every resolved address; hostname tier extracts the host part
correctly for [v6]/host:port/bare-v6 forms. 44 tests.

* test: pin 404 handler dual-branch behavior (browser theme page vs API JSON)

* fix: write back normalized endpoint values; hostname dev-gate test

* test: header-injection negative paths for attachment disposition

* fix: S3 merge buffers parts to >=5MiB; missing object 404 upfront

- S3 multipart rejects parts <5MiB (EntityTooSmall) except the last; chunk
  size is client-controlled (commonly 2-4MB) so multi-chunk merges always
  failed with 500. Merge now buffers chunks into >=5MiB parts (memory
  bound: 5MB + one chunk).
- get_file_response: a 404 head_object now raises StorageError(404)
  upfront instead of signing a doomed presigned URL and failing mid-stream
  (aligns with local backend semantics from the M2 behavior unification).

* test: S3 backend coverage via in-process moto server (13 cases)

mock_aws cannot intercept aioboto3's aiohttp stack, so tests run against
a real in-process ThreadedMotoServer endpoint. Covers roundtrips, missing
object 404, merge failure modes (missing chunk / hash mismatch -> abort,
no leftover object), cleanup scoping, presign shapes, proxy dispatch.
Also adds moto[s3,server] to the dev group and CI.

* fix: WebDAV missing object 404 upfront; connection errors map to 503

HEAD 404 was silently swallowed (signed a 200 bad stream that failed
mid-download, same family as the S3 bug), and HEAD connection errors were
also swallowed so the 503 mapping never fired. Both now surface properly;
the aiohttp session is reclaimed on pre-stream exception paths.

* test: WebDAV backend coverage via in-process WebDAV server (9 cases)

Real HTTP (HEAD/GET/PUT/DELETE/MKCOL/PROPFIND) against a minimal aiohttp
WebDAV server over a temp dir: roundtrips, missing object 404, connection
error 503, delete with empty-parent cleanup, merge failure modes, chunk
cleanup scoping.

* fix: OneDrive missing object maps to 404 upfront; OpenDAL behavior pinned

OneDrive get_file_response translated graph itemNotFound into the outer
503 catch-all; now raises StorageError(404) like local/S3/WebDAV.
OpenDAL already mapped missing objects to 404 via its outer catch —
pinned with fake-operator tests (SDK not in runtime deps, instances built
via __new__). Also tightens the S3 presign URL assertion.

* test: admin write-path coverage (batch/single delete, batch update, policy actions)

13 negative-path cases for the previously untested data-modifying admin
endpoints: mixed-id aggregation with duplicates and missing records,
empty-list rejections, clear_expired_at permanence semantics, policy
action boundaries (zero limit 400, unknown action 400, missing 404),
and the DoesNotExist->404 mapping pinned for single delete.

* test: admin read-path and view-preset CRUD coverage (22 cases)

Detail/metadata/preview/admin-download/activities/local-lists+delete/
verify: missing-record 404s, note/tag truncation limits, preview max_chars
truncation, activity filtering and the 80-event clamp, local path
traversal rejection. Presets: update-vs-create by id, name truncation,
filter normalization clamps, 24-preset cap, delete-missing 404.

* refactor: move auth primitives to apps/base/auth, kill base->admin dep

JWT create/verify, bearer extraction and the share-upload gate move to
apps.base.auth (consumed by both surfaces); admin.dependencies keeps
admin session gating and service providers, re-exporting the primitives
for import-path compatibility. base.views no longer imports from
apps.admin — the last cross-app reverse dependency is gone.

* refactor: split admin services god-module (ConfigService/LocalFile out)

apps/admin/services.py (1697 lines) becomes FileService (~1200 lines) +
a compatibility facade re-exporting ConfigService, LocalFileService,
LocalFileClass and keyvalue_write_lock from their new homes
(config_service.py holds the D5 KeyValue lock; one-way imports, no
cycle). test_admin_security patch targets follow ConfigService to its
new module. Behavior unchanged; 247 tests green.

* fix: size-less uploads no longer crash (upstream seek bug)

validate_file_size's size-None branch called UploadFile.seek(0, 2),
which raises TypeError (UploadFile.seek takes one arg); the underlying
SpooledTemporaryFile.seek is sync, so awaiting it also fails. Use the
underlying file object with sync seek(0, SEEK_END). Surface discovered
by the mypy pilot run; regression test covers the size-None branch.

* chore: bump direct deps (patch/minor)

fastapi 0.139.2->0.141.1, pydantic 2.12.5->2.13.5, uvicorn 0.51.0->0.53.0,
aiohttp 3.14.2->3.14.3; lockfile regenerated. tortoise-orm 0.x->1.x major
deliberately deferred pending API-change review.

* ci: gradual mypy adoption via baseline ratchet

scripts/mypy_ratchet.py compares mypy output against
scripts/mypy-baseline.txt (line numbers stripped for edit stability):
new type errors fail CI, fixes are folded back via --regenerate.
Baseline starts at 30 known errors; the seek bug found in the pilot was
fixed before baselining. mypy joins the dev group and CI.

* fix: migrate WebDAV auth off aiohttp.BasicAuth (deprecated in 4.0)

Dependency-upgrade probe on aiohttp 3.14.3 surfaced the BasicAuth
deprecation; switch to aiohttp.encode_basic_auth() headers so the
aiohttp 4.0 upgrade does not break the WebDAV backend.

* chore: drop dead code (WebDAV _instance stub, unused LocalFileClass.write)

* test: file-validation negative paths (13 cases)

Magic-bytes spoofing (text-as-png, exe-as-pdf, mp4/webp box detection),
whitelist rule semantics (star/image-wildcard/content mismatch), chunk-0
header validation parity, and end-to-end UploadFile rejection.

* test: migration runner coverage (5 cases)

Full chain executes 001-007 in filename order and registers; rerun is
idempotent; pre-registered entries are skipped (DDL not re-executed);
the resulting schema matches deployment; a failing migration propagates
and is not registered. Uses a fresh :memory: DB without
generate_schemas so the migrations themselves build the schema (the
real deployment path).

* refactor: split core/storage.py into a package (per-backend modules)

core/storage.py (1400+ lines, six backends) becomes core/storage/ with
_base.py (data contracts, interface, shared header builder), and
local/s3/onedrive/opendal/webdav modules. The package __init__
re-exports the full historical public surface; test patch targets move
to the precise backend modules (core.storage.local.data_root etc.) and
the attachment guard scans the package. Import-only change; 264 tests
green. Storage-layer DI deferred: current tests construct instances
against patched settings, so the added surface isn't justified yet.

* refactor: TypedDict for IPRateLimit records

The rate-limit ledger mixed int and datetime under a loose Union typing,
producing 5 mypy noise errors. A TypedDict gives precise per-key types;
setdefault replaces the get-then-assign pattern. One mypy baseline error
family resolved; 264 tests green.

* chore: tighten mypy baseline after IPRateLimit TypedDict

* refactor: resolve all 26 baseline mypy errors; precise test assertions

Per-error treatment (no bulk script): honest annotations for
StoredDownload (Path/bytes/Callable), KeyValue.value (Any for JSONField),
dict[str, Any] for form/config dicts; signature split in create_token;
str() normalization in config int(); platform/type-ignore only where the
checker cannot follow (msvcrt, FastAPI Request injection). Baseline file
and ratchet compare now redundant at zero errors but kept as the guard
against regressions.

Tests: assertions tightened to status_code == 403 with HTTPException
raises; _create_share explicit parameters replace **extra passthrough;
seek-fix pointer-reset assertion persisted.
2026-09-20 03:01:08 +09:00

416 lines
15 KiB
Python

"""apps.base service layer: upload orchestration shared by share/chunk/presign.
Handlers in views.py keep HTTP concerns (form parsing, session lookup, status
codes for malformed requests, rate limiting); this module owns the business
workflow: quota reservation, storage writes, share-record creation, and
failure rollback/cleanup.
"""
import os
from fastapi import HTTPException, UploadFile
from fastapi.responses import FileResponse, Response, StreamingResponse
from core.logger import logger
from core.settings import settings
from core.storage import FileStorageInterface, StoredDownload, StoredFile, storages
from apps.base.file_validation import validate_upload_file
from apps.base.local_share import build_local_ref_download, is_local_ref
from apps.base.models import FileCodes, PresignUploadSession, UploadChunk
from apps.base.quota import release_storage, reserve_storage
from apps.base.utils import build_file_path, get_expire_info
import uuid
# 预签名上传会话有效期(秒)
PRESIGN_SESSION_EXPIRES = 900 # 15分钟
def stored_file_of(code: FileCodes) -> StoredFile:
"""Adapter: project an ORM FileCodes row onto the storage-layer contract."""
return StoredFile(
file_path=code.file_path,
uuid_file_name=code.uuid_file_name,
code=code.code,
prefix=code.prefix,
suffix=code.suffix,
text=code.text or "",
)
async def get_stored_download(file_code, file_storage: FileStorageInterface | None = None) -> StoredDownload:
if is_local_ref(file_code):
return build_local_ref_download(file_code)
storage = file_storage or storages[settings.file_storage]()
return await storage.get_file_response(stored_file_of(file_code))
async def rollback_saved_file(
storage: FileStorageInterface,
file_path: str,
uuid_file_name: str,
*,
context: str,
upload_id: str | None = None,
) -> None:
"""Best-effort delete of a stored file after a failed upload workflow.
Rollback failures are logged (with traceback) and swallowed: the original
error must propagate; the file, if left behind, is cleaned by the expiry
task once its orphaned record ages out.
"""
try:
await storage.delete_file(
StoredFile(file_path=file_path, uuid_file_name=uuid_file_name)
)
except Exception:
logger.warning(
"%s:回滚删除已保存文件失败%s",
context,
f" upload_id={upload_id}" if upload_id else "",
exc_info=True,
)
async def validate_file_size(file: UploadFile, max_size: int) -> int:
"""Return the upload's size, rejecting anything above max_size."""
size = file.size
if size is None:
# 无长度声明的上传(如 chunked 传输):必须走底层文件对象拿末尾偏移。
# 注意两层都不能用:UploadFile.seek 只接受单参数(TypeError);
# 底层 SpooledTemporaryFile.seek 是同步方法(不能 await)。
file.file.seek(0, 2)
size = file.file.tell()
file.file.seek(0)
if size > max_size:
max_size_mb = max_size / (1024 * 1024)
raise HTTPException(
status_code=403, detail=f"大小超过限制,最大为{max_size_mb:.2f} MB"
)
return size
def chunk_reservation_ttl() -> int:
ttl = max(1, int(getattr(settings, "chunk_expire_hours", 24))) * 3600
return ttl
class FileUploadService:
"""统一的文件上传服务"""
@staticmethod
def _storage() -> FileStorageInterface:
return storages[settings.file_storage]()
@staticmethod
async def generate_file_path(
file_name: str, upload_id: str | None = None
) -> tuple[str, str, str, str, str]:
"""Delegates path generation to apps.base.utils.build_file_path."""
return await build_file_path(file_name, upload_id or uuid.uuid4().hex)
@staticmethod
async def create_file_record(
file_name: str,
file_size: int,
file_path: str,
expire_value: int,
expire_style: str,
**extra_fields,
) -> str:
"""统一创建FileCodes记录,返回code"""
expired_at, expired_count, used_count, code = await get_expire_info(
expire_value, expire_style
)
prefix, suffix = os.path.splitext(file_name)
await FileCodes.create(
code=code,
prefix=prefix,
suffix=suffix,
uuid_file_name=file_name,
file_path=file_path,
size=file_size,
expired_at=expired_at,
expired_count=expired_count,
used_count=used_count,
**extra_fields,
)
return code
@staticmethod
async def create_text_share(
text: str, expire_value: int, expire_style: str
) -> str:
"""文本分享:配额预留 → 建分享记录 → 释放配额。"""
text_size = len(text.encode("utf-8"))
token = f"text:{uuid.uuid4().hex}"
await reserve_storage(token, text_size, ttl_seconds=300)
try:
expired_at, expired_count, used_count, code = await get_expire_info(
expire_value, expire_style
)
await FileCodes.create(
code=code,
text=text,
expired_at=expired_at,
expired_count=expired_count,
used_count=used_count,
size=text_size,
prefix="Text",
)
finally:
await release_storage(token)
return code
@staticmethod
async def create_file_share(
file: UploadFile, *, size: int, expire_value: int, expire_style: str
) -> dict[str, str]:
"""文件分享:路径生成 → 配额预留 → 存储写入 → 建分享记录,失败回滚已存文件。"""
path, suffix, prefix, uuid_file_name, save_path = (
await FileUploadService.generate_file_path(file.filename or "")
)
token = f"file:{uuid.uuid4().hex}"
await reserve_storage(token, size, ttl_seconds=3600)
storage = FileUploadService._storage()
try:
expired_at, expired_count, used_count, code = await get_expire_info(
expire_value, expire_style
)
await storage.save_file(file.file, save_path, file.content_type)
await FileCodes.create(
code=code,
prefix=prefix,
suffix=suffix,
uuid_file_name=uuid_file_name,
file_path=path,
size=size,
expired_at=expired_at,
expired_count=expired_count,
used_count=used_count,
)
except Exception:
await rollback_saved_file(
storage, path, uuid_file_name, context="分享上传"
)
raise
finally:
await release_storage(token)
# UploadFile.filename 类型为 str | None;路径生成已用 "" 兜底,
# 响应处保持 str 以匹配 dict[str, str] 契约
return {"code": code, "name": file.filename or ""}
@staticmethod
async def complete_chunked_upload(
upload_id: str, chunk_info: UploadChunk, expire_value: int, expire_style: str
) -> dict[str, str]:
"""分片合并:配额 → 完整性/大小校验 → 合并 → 建分享记录 → 清理分片。
失败路径的配额释放与清理范围与原实现逐一对齐:
完整性校验失败仅抛 400(预留由 TTL 兜底);合并失败清理分片文件后抛 500。
"""
storage = FileUploadService._storage()
await reserve_storage(
f"chunk:{upload_id}", chunk_info.file_size, ttl_seconds=chunk_reservation_ttl()
)
completed_chunks = await UploadChunk.filter(
upload_id=upload_id, completed=True
).all()
if len(completed_chunks) != chunk_info.total_chunks:
raise HTTPException(400, "分片不完整")
# 用分片数 * chunk_size 校验最大可能大小
max_total_size = len(completed_chunks) * chunk_info.chunk_size
if max_total_size > settings.upload_size:
save_path = chunk_info.save_path
if save_path:
try:
await storage.clean_chunks(upload_id, save_path)
except Exception:
logger.warning(
"分片超限中止:清理分片文件失败 upload_id=%s",
upload_id,
exc_info=True,
)
await UploadChunk.filter(upload_id=upload_id).delete()
await release_storage(f"chunk:{upload_id}")
max_size_mb = settings.upload_size / (1024 * 1024)
raise HTTPException(
403, f"实际上传大小超过限制,最大为 {max_size_mb:.2f} MB"
)
save_path = chunk_info.save_path
path = os.path.dirname(save_path) if save_path else ""
safe_file_name = os.path.basename(save_path) if save_path else ""
prefix, suffix = os.path.splitext(safe_file_name)
try:
records = {r.chunk_index: r for r in completed_chunks}
_, file_hash = await storage.merge_chunks(
upload_id,
chunk_info.total_chunks,
chunk_info.chunk_size,
save_path,
records,
)
expired_at, expired_count, used_count, code = await get_expire_info(
expire_value, expire_style
)
await FileCodes.create(
code=code,
file_hash=file_hash, # 使用合并后计算的哈希
is_chunked=True,
upload_id=upload_id,
size=chunk_info.file_size,
expired_at=expired_at,
expired_count=expired_count,
used_count=used_count,
file_path=path,
uuid_file_name=safe_file_name,
prefix=prefix,
suffix=suffix,
)
await storage.clean_chunks(upload_id, save_path)
await UploadChunk.filter(upload_id=upload_id).delete()
await release_storage(f"chunk:{upload_id}")
return {"code": code, "name": safe_file_name}
except ValueError as e:
raise HTTPException(400, str(e))
except Exception as e:
# 合并失败时清理临时文件
try:
await storage.clean_chunks(upload_id, save_path)
except Exception:
logger.warning(
"分片合并失败:清理临时分片文件失败 upload_id=%s",
upload_id,
exc_info=True,
)
raise HTTPException(500, f"文件合并失败: {str(e)}")
@staticmethod
async def commit_proxy_upload(
session: PresignUploadSession, file: UploadFile
) -> str:
"""预签名代理上传:配额 → 大小/类型/一致性校验 → 转存 → 建记录 → 会话清理。
校验失败不释放预留(与原实现一致,由 TTL 兜底)。
"""
await reserve_storage(
f"presign:{session.upload_id}",
session.file_size,
ttl_seconds=PRESIGN_SESSION_EXPIRES,
)
file_size = await validate_file_size(file, settings.upload_size)
await validate_upload_file(file)
if abs(file_size - session.file_size) > 1024:
raise HTTPException(400, "文件大小与声明不符")
storage = FileUploadService._storage()
try:
await storage.save_file(file.file, session.save_path, file.content_type)
except Exception as e:
raise HTTPException(500, f"文件保存失败: {str(e)}")
try:
code = await FileUploadService.create_file_record(
session.file_name,
file_size,
os.path.dirname(session.save_path),
session.expire_value,
session.expire_style,
)
except Exception:
await rollback_saved_file(
storage,
os.path.dirname(session.save_path),
os.path.basename(session.save_path),
context="预签名代理上传:记录创建失败",
upload_id=session.upload_id,
)
raise
await session.delete()
await release_storage(f"presign:{session.upload_id}")
return code
@staticmethod
async def confirm_direct_upload(session: PresignUploadSession) -> str:
"""预签名直传确认:配额 → 文件存在性 → 建记录 → 会话清理。
预留失败说明配额已耗尽,此时清理远端临时文件与会话后原样抛出。
"""
try:
await reserve_storage(
f"presign:{session.upload_id}",
session.file_size,
ttl_seconds=PRESIGN_SESSION_EXPIRES,
)
except HTTPException:
storage = FileUploadService._storage()
try:
if await storage.file_exists(session.save_path):
await storage.delete_file(
StoredFile(
file_path=os.path.dirname(session.save_path),
uuid_file_name=os.path.basename(session.save_path),
)
)
finally:
await session.delete()
await release_storage(f"presign:{session.upload_id}")
raise
storage = FileUploadService._storage()
if not await storage.file_exists(session.save_path):
raise HTTPException(404, "文件未上传或上传失败")
try:
code = await FileUploadService.create_file_record(
session.file_name,
session.file_size,
os.path.dirname(session.save_path),
session.expire_value,
session.expire_style,
)
except Exception:
await rollback_saved_file(
storage,
os.path.dirname(session.save_path),
os.path.basename(session.save_path),
context="预签名确认:记录创建失败",
upload_id=session.upload_id,
)
raise
await session.delete()
await release_storage(f"presign:{session.upload_id}")
return code
def response_from_download(download: StoredDownload):
"""Build the starlette Response for a StoredDownload (view-layer duty)."""
if download.path is not None:
return FileResponse(
download.path,
media_type=download.media_type,
headers=download.headers,
filename=download.filename,
)
if download.content is not None:
return Response(
download.content, media_type=download.media_type, headers=download.headers
)
if download.stream_factory is None:
# 数据契约要求三者至少其一;防御性 500 语义
raise ValueError("StoredDownload 缺少可用的下载载荷")
return StreamingResponse(
download.stream_factory(),
media_type=download.media_type,
headers=download.headers,
background=download.background,
)