mirror of
https://github.com/vastsa/FileCodeBox.git
synced 2026-11-20 13:29:56 +08:00
* fix: constant-time comparison for download tokens (security item 5) Replace the set-membership key check in /share/download with hmac.compare_digest over the two valid window tokens; add negative-path tests (wrong key 403, both windows accepted, foreign-code token 403). * fix: single source for attachment headers + guard tests (security item 2) All six Content-Disposition construction sites across the five storage backends now go through build_attachment_headers; the header is what neutralizes stored XSS on the same-origin download path. Guard tests assert every get_file_response uses the builder and no hand-built disposition reappears. * fix: reject internal-network endpoints for storage config (security item 7) s3_endpoint_url/s3_hostname/webdav_url are fetched server-side; with APP_ENV=production the write entry now enforces http(s) schemes and a loopback/private/link-local/metadata host blacklist. Development env stays open for local minio/webdav. Already-stored values are never re-validated, so existing deployments are unaffected. * fix: JSON 404 for handler-raised 404s; negative-path test batch Starlette status-code handlers take precedence over the HTTPException class handler, so registering the theme index as the 404 handler turned every HTTPException(404) app-wide into a 200 HTML page (found by the new negative-path tests). The new not_found_handler serves the theme page only to browsers (Accept: text/html) and JSON 404 to API clients. Negative-path batch: download count exhaustion, missing chunk session 404s, expired presign session deletion, admin update_file uniqueness/ existence. * fix: run container as non-root app user with compat volume chown (security item 3) Entrypoint starts as root only to fix the data-dir ownership (skipped when nothing needs changing or when the deployer sets an explicit user), then drops to uid 10001 via gosu. Verified in-container: uvicorn runs as app (uid 10001), setup and share round-trip work against a root-owned volume. * fix: compare download tokens as bytes (non-ASCII key caused 500) * test: lock changed-only semantics for endpoint config saves Integration regression for the background lesson: a stored internal endpoint must not block unrelated settings saves in production mode; changing to an internal endpoint (URL or bare-hostname form) is 400. * fix: resolve-based SSRF check — deny IP shorthands, hex/decimal IPs, DNS maps Probe found the static blacklist bypassed by glibc shorthands (127.1, 10.1), decimal (2130706433) and hex IP forms, *.localhost, and loopback-mapping DNS services. Both validators now resolve the host and re-check every resolved address; hostname tier extracts the host part correctly for [v6]/host:port/bare-v6 forms. 44 tests. * test: pin 404 handler dual-branch behavior (browser theme page vs API JSON) * fix: write back normalized endpoint values; hostname dev-gate test * test: header-injection negative paths for attachment disposition * fix: S3 merge buffers parts to >=5MiB; missing object 404 upfront - S3 multipart rejects parts <5MiB (EntityTooSmall) except the last; chunk size is client-controlled (commonly 2-4MB) so multi-chunk merges always failed with 500. Merge now buffers chunks into >=5MiB parts (memory bound: 5MB + one chunk). - get_file_response: a 404 head_object now raises StorageError(404) upfront instead of signing a doomed presigned URL and failing mid-stream (aligns with local backend semantics from the M2 behavior unification). * test: S3 backend coverage via in-process moto server (13 cases) mock_aws cannot intercept aioboto3's aiohttp stack, so tests run against a real in-process ThreadedMotoServer endpoint. Covers roundtrips, missing object 404, merge failure modes (missing chunk / hash mismatch -> abort, no leftover object), cleanup scoping, presign shapes, proxy dispatch. Also adds moto[s3,server] to the dev group and CI. * fix: WebDAV missing object 404 upfront; connection errors map to 503 HEAD 404 was silently swallowed (signed a 200 bad stream that failed mid-download, same family as the S3 bug), and HEAD connection errors were also swallowed so the 503 mapping never fired. Both now surface properly; the aiohttp session is reclaimed on pre-stream exception paths. * test: WebDAV backend coverage via in-process WebDAV server (9 cases) Real HTTP (HEAD/GET/PUT/DELETE/MKCOL/PROPFIND) against a minimal aiohttp WebDAV server over a temp dir: roundtrips, missing object 404, connection error 503, delete with empty-parent cleanup, merge failure modes, chunk cleanup scoping. * fix: OneDrive missing object maps to 404 upfront; OpenDAL behavior pinned OneDrive get_file_response translated graph itemNotFound into the outer 503 catch-all; now raises StorageError(404) like local/S3/WebDAV. OpenDAL already mapped missing objects to 404 via its outer catch — pinned with fake-operator tests (SDK not in runtime deps, instances built via __new__). Also tightens the S3 presign URL assertion. * test: admin write-path coverage (batch/single delete, batch update, policy actions) 13 negative-path cases for the previously untested data-modifying admin endpoints: mixed-id aggregation with duplicates and missing records, empty-list rejections, clear_expired_at permanence semantics, policy action boundaries (zero limit 400, unknown action 400, missing 404), and the DoesNotExist->404 mapping pinned for single delete. * test: admin read-path and view-preset CRUD coverage (22 cases) Detail/metadata/preview/admin-download/activities/local-lists+delete/ verify: missing-record 404s, note/tag truncation limits, preview max_chars truncation, activity filtering and the 80-event clamp, local path traversal rejection. Presets: update-vs-create by id, name truncation, filter normalization clamps, 24-preset cap, delete-missing 404. * refactor: move auth primitives to apps/base/auth, kill base->admin dep JWT create/verify, bearer extraction and the share-upload gate move to apps.base.auth (consumed by both surfaces); admin.dependencies keeps admin session gating and service providers, re-exporting the primitives for import-path compatibility. base.views no longer imports from apps.admin — the last cross-app reverse dependency is gone. * refactor: split admin services god-module (ConfigService/LocalFile out) apps/admin/services.py (1697 lines) becomes FileService (~1200 lines) + a compatibility facade re-exporting ConfigService, LocalFileService, LocalFileClass and keyvalue_write_lock from their new homes (config_service.py holds the D5 KeyValue lock; one-way imports, no cycle). test_admin_security patch targets follow ConfigService to its new module. Behavior unchanged; 247 tests green. * fix: size-less uploads no longer crash (upstream seek bug) validate_file_size's size-None branch called UploadFile.seek(0, 2), which raises TypeError (UploadFile.seek takes one arg); the underlying SpooledTemporaryFile.seek is sync, so awaiting it also fails. Use the underlying file object with sync seek(0, SEEK_END). Surface discovered by the mypy pilot run; regression test covers the size-None branch. * chore: bump direct deps (patch/minor) fastapi 0.139.2->0.141.1, pydantic 2.12.5->2.13.5, uvicorn 0.51.0->0.53.0, aiohttp 3.14.2->3.14.3; lockfile regenerated. tortoise-orm 0.x->1.x major deliberately deferred pending API-change review. * ci: gradual mypy adoption via baseline ratchet scripts/mypy_ratchet.py compares mypy output against scripts/mypy-baseline.txt (line numbers stripped for edit stability): new type errors fail CI, fixes are folded back via --regenerate. Baseline starts at 30 known errors; the seek bug found in the pilot was fixed before baselining. mypy joins the dev group and CI. * fix: migrate WebDAV auth off aiohttp.BasicAuth (deprecated in 4.0) Dependency-upgrade probe on aiohttp 3.14.3 surfaced the BasicAuth deprecation; switch to aiohttp.encode_basic_auth() headers so the aiohttp 4.0 upgrade does not break the WebDAV backend. * chore: drop dead code (WebDAV _instance stub, unused LocalFileClass.write) * test: file-validation negative paths (13 cases) Magic-bytes spoofing (text-as-png, exe-as-pdf, mp4/webp box detection), whitelist rule semantics (star/image-wildcard/content mismatch), chunk-0 header validation parity, and end-to-end UploadFile rejection. * test: migration runner coverage (5 cases) Full chain executes 001-007 in filename order and registers; rerun is idempotent; pre-registered entries are skipped (DDL not re-executed); the resulting schema matches deployment; a failing migration propagates and is not registered. Uses a fresh :memory: DB without generate_schemas so the migrations themselves build the schema (the real deployment path). * refactor: split core/storage.py into a package (per-backend modules) core/storage.py (1400+ lines, six backends) becomes core/storage/ with _base.py (data contracts, interface, shared header builder), and local/s3/onedrive/opendal/webdav modules. The package __init__ re-exports the full historical public surface; test patch targets move to the precise backend modules (core.storage.local.data_root etc.) and the attachment guard scans the package. Import-only change; 264 tests green. Storage-layer DI deferred: current tests construct instances against patched settings, so the added surface isn't justified yet. * refactor: TypedDict for IPRateLimit records The rate-limit ledger mixed int and datetime under a loose Union typing, producing 5 mypy noise errors. A TypedDict gives precise per-key types; setdefault replaces the get-then-assign pattern. One mypy baseline error family resolved; 264 tests green. * chore: tighten mypy baseline after IPRateLimit TypedDict * refactor: resolve all 26 baseline mypy errors; precise test assertions Per-error treatment (no bulk script): honest annotations for StoredDownload (Path/bytes/Callable), KeyValue.value (Any for JSONField), dict[str, Any] for form/config dicts; signature split in create_token; str() normalization in config int(); platform/type-ignore only where the checker cannot follow (msvcrt, FastAPI Request injection). Baseline file and ratchet compare now redundant at zero errors but kept as the guard against regressions. Tests: assertions tightened to status_code == 403 with HTTPException raises; _create_share explicit parameters replace **extra passthrough; seek-fix pointer-reset assertion persisted.
693 lines
25 KiB
Python
693 lines
25 KiB
Python
import hashlib
|
|
import hmac
|
|
import os
|
|
import uuid
|
|
from datetime import timedelta
|
|
from urllib.parse import quote, unquote
|
|
|
|
from typing import Optional, Tuple, Union
|
|
|
|
from fastapi import APIRouter, Form, Request, UploadFile, File, Depends, HTTPException
|
|
from pydantic import BaseModel, ValidationError
|
|
from starlette import status
|
|
from starlette.responses import Response
|
|
from tortoise.expressions import Case, F, Q, When
|
|
|
|
from apps.base.auth import share_required_login
|
|
from apps.base.models import FileCodes, UploadChunk, PresignUploadSession
|
|
from apps.base.quota import release_storage, reserve_storage
|
|
from apps.base.services import (
|
|
PRESIGN_SESSION_EXPIRES,
|
|
FileUploadService,
|
|
get_stored_download,
|
|
response_from_download,
|
|
stored_file_of,
|
|
validate_file_size,
|
|
)
|
|
from core.storage import StoredFile
|
|
from core.logger import logger
|
|
from apps.base.schemas import (
|
|
SelectFileModel,
|
|
InitChunkUploadModel,
|
|
CompleteUploadModel,
|
|
PresignUploadInitRequest,
|
|
)
|
|
from apps.base.file_validation import validate_file_type, validate_upload_file, validate_header_bytes
|
|
from apps.base.utils import (
|
|
ip_limit,
|
|
get_chunk_file_path_name,
|
|
validate_expire_style,
|
|
)
|
|
from apps.base.local_share import is_local_ref
|
|
from core.response import APIResponse
|
|
from core.settings import settings
|
|
from core.storage import storages, FileStorageInterface
|
|
from core.utils import (
|
|
get_file_url as get_proxy_file_url,
|
|
get_select_token,
|
|
get_now,
|
|
sanitize_filename,
|
|
)
|
|
|
|
share_api = APIRouter(prefix="/share", tags=["分享"])
|
|
|
|
|
|
# ============ 公共服务层 ============
|
|
|
|
|
|
def normalize_share_code(code: str) -> str:
|
|
return str(code or "").strip()
|
|
|
|
|
|
@share_api.post("/text/", dependencies=[Depends(share_required_login)])
|
|
async def share_text(
|
|
text: str = Form(...),
|
|
expire_value: int = Form(default=1, gt=0),
|
|
expire_style: str = Form(default="day"),
|
|
ip: str = Depends(ip_limit["upload"]),
|
|
):
|
|
validate_expire_style(expire_style)
|
|
text_size = len(text.encode("utf-8"))
|
|
max_txt_size = 222 * 1024
|
|
if text_size > max_txt_size:
|
|
raise HTTPException(status_code=403, detail="内容过多,建议采用文件形式")
|
|
|
|
code = await FileUploadService.create_text_share(text, expire_value, expire_style)
|
|
ip_limit["upload"].add_ip(ip)
|
|
return APIResponse(detail={"code": code})
|
|
|
|
|
|
@share_api.post("/file/", dependencies=[Depends(share_required_login)])
|
|
async def share_file(
|
|
expire_value: int = Form(default=1, gt=0),
|
|
expire_style: str = Form(default="day"),
|
|
file: UploadFile = File(...),
|
|
ip: str = Depends(ip_limit["upload"]),
|
|
):
|
|
file_size = await validate_file_size(file, settings.upload_size)
|
|
await validate_upload_file(file)
|
|
validate_expire_style(expire_style)
|
|
detail = await FileUploadService.create_file_share(
|
|
file, size=file_size, expire_value=expire_value, expire_style=expire_style
|
|
)
|
|
ip_limit["upload"].add_ip(ip)
|
|
return APIResponse(detail=detail)
|
|
|
|
|
|
async def get_code_file_by_code(
|
|
code: str, check: bool = True
|
|
) -> Tuple[bool, Union[FileCodes, str]]:
|
|
normalized_code = normalize_share_code(code)
|
|
if not normalized_code:
|
|
return False, "文件不存在"
|
|
file_code = await FileCodes.filter(code=normalized_code).first()
|
|
if not file_code:
|
|
return False, "文件不存在"
|
|
if await file_code.is_expired() and check:
|
|
return False, "文件已过期"
|
|
return True, file_code
|
|
|
|
|
|
async def consume_file_usage(file_code: FileCodes) -> bool:
|
|
"""原子校验分享状态并记录一次实际领取。"""
|
|
now = await get_now()
|
|
eligible = (
|
|
Q(expired_count__gt=0)
|
|
| Q(expired_count__lt=0, expired_at__gt=now)
|
|
| Q(expired_count__lt=0, expired_at=None)
|
|
)
|
|
updated = await FileCodes.filter(Q(id=file_code.id) & eligible).update(
|
|
expired_count=Case(
|
|
When(expired_count__gt=0, then=F("expired_count") - 1),
|
|
default=F("expired_count"),
|
|
),
|
|
used_count=F("used_count") + 1,
|
|
)
|
|
if not updated:
|
|
return False
|
|
await file_code.refresh_from_db()
|
|
return True
|
|
|
|
|
|
def build_file_metadata(file_code: FileCodes) -> dict:
|
|
is_text = file_code.text is not None
|
|
remaining_downloads = (
|
|
file_code.expired_count if file_code.expired_count > 0 else None
|
|
)
|
|
return {
|
|
"code": file_code.code,
|
|
"name": file_code.prefix + file_code.suffix,
|
|
"size": file_code.size,
|
|
"type": "text" if is_text else "file",
|
|
"is_text": is_text,
|
|
"created_at": file_code.created_at,
|
|
"expired_at": file_code.expired_at,
|
|
"expired_count": file_code.expired_count,
|
|
"used_count": file_code.used_count,
|
|
"remaining_downloads": remaining_downloads,
|
|
}
|
|
|
|
|
|
async def build_select_detail(
|
|
file_code: FileCodes, file_storage: FileStorageInterface
|
|
) -> dict:
|
|
metadata = build_file_metadata(file_code)
|
|
if file_code.text is not None:
|
|
download_url = None
|
|
elif file_code.expired_count >= 0 or is_local_ref(file_code):
|
|
# 次数限制与 NAS 引用都必须走下载接口,避免直链绕过次数或打到错误存储。
|
|
download_url = await get_proxy_file_url(file_code.code)
|
|
else:
|
|
download_url = await file_storage.get_file_url(stored_file_of(file_code))
|
|
content = file_code.text if file_code.text is not None else None
|
|
return {
|
|
**metadata,
|
|
"text": content if content is not None else download_url,
|
|
"content": content,
|
|
"download_url": download_url,
|
|
}
|
|
|
|
|
|
@share_api.get("/metadata/")
|
|
async def get_file_metadata(code: str, ip: str = Depends(ip_limit["metadata"])):
|
|
has, file_code = await get_code_file_by_code(code)
|
|
if not has:
|
|
ip_limit["metadata"].add_ip(ip)
|
|
return APIResponse(code=404, detail=file_code)
|
|
|
|
assert isinstance(file_code, FileCodes)
|
|
ip_limit["metadata"].add_ip(ip)
|
|
return APIResponse(detail=build_file_metadata(file_code))
|
|
|
|
|
|
@share_api.post("/metadata/")
|
|
async def post_file_metadata(
|
|
data: SelectFileModel, ip: str = Depends(ip_limit["metadata"])
|
|
):
|
|
has, file_code = await get_code_file_by_code(data.code)
|
|
if not has:
|
|
ip_limit["metadata"].add_ip(ip)
|
|
return APIResponse(code=404, detail=file_code)
|
|
|
|
assert isinstance(file_code, FileCodes)
|
|
ip_limit["metadata"].add_ip(ip)
|
|
return APIResponse(detail=build_file_metadata(file_code))
|
|
|
|
|
|
@share_api.get("/select/")
|
|
async def get_code_file(code: str, ip: str = Depends(ip_limit["error"])):
|
|
file_storage: FileStorageInterface = storages[settings.file_storage]()
|
|
has, file_code = await get_code_file_by_code(code)
|
|
if not has:
|
|
ip_limit["error"].add_ip(ip)
|
|
return APIResponse(code=404, detail=file_code)
|
|
|
|
assert isinstance(file_code, FileCodes)
|
|
if not await consume_file_usage(file_code):
|
|
return APIResponse(code=404, detail="文件已过期")
|
|
if file_code.text is not None:
|
|
filename = f"{file_code.prefix or 'Text'}{file_code.suffix or '.txt'}"
|
|
return Response(
|
|
content=file_code.text,
|
|
media_type="text/plain",
|
|
headers={
|
|
"Content-Disposition": (
|
|
f"attachment; filename*=UTF-8''{quote(filename, safe='')}"
|
|
)
|
|
},
|
|
)
|
|
return response_from_download(await get_stored_download(file_code, file_storage))
|
|
|
|
|
|
@share_api.post("/select/")
|
|
async def select_file(data: SelectFileModel, ip: str = Depends(ip_limit["error"])):
|
|
file_storage: FileStorageInterface = storages[settings.file_storage]()
|
|
has, file_code = await get_code_file_by_code(data.code)
|
|
if not has:
|
|
ip_limit["error"].add_ip(ip)
|
|
return APIResponse(code=404, detail=file_code)
|
|
|
|
assert isinstance(file_code, FileCodes)
|
|
detail = await build_select_detail(file_code, file_storage)
|
|
download_url = detail.get("download_url")
|
|
consumes_on_download = isinstance(download_url, str) and download_url.startswith(
|
|
"/share/download?"
|
|
)
|
|
if not consumes_on_download and not await consume_file_usage(file_code):
|
|
return APIResponse(code=404, detail="文件已过期")
|
|
if not consumes_on_download:
|
|
detail.update(build_file_metadata(file_code))
|
|
return APIResponse(detail=detail)
|
|
|
|
|
|
@share_api.get("/download")
|
|
async def download_file(key: str, code: str, ip: str = Depends(ip_limit["error"])):
|
|
file_storage: FileStorageInterface = storages[settings.file_storage]()
|
|
normalized_code = normalize_share_code(code)
|
|
# 同时接受当前窗口与上一窗口 token,避免时间窗边界竞态导致偶发 403。
|
|
# 逐个常量时间比较(hmac.compare_digest),避免 set 成员判断的时序侧信道。
|
|
valid_keys = [
|
|
await get_select_token(normalized_code, offset=0),
|
|
await get_select_token(normalized_code, offset=1),
|
|
]
|
|
if not any(
|
|
hmac.compare_digest(key.encode(), candidate.encode())
|
|
for candidate in valid_keys
|
|
):
|
|
ip_limit["error"].add_ip(ip)
|
|
raise HTTPException(status_code=403, detail="下载鉴权失败")
|
|
has, file_code = await get_code_file_by_code(normalized_code)
|
|
if not has:
|
|
return APIResponse(code=404, detail=file_code)
|
|
assert isinstance(file_code, FileCodes)
|
|
if not await consume_file_usage(file_code):
|
|
return APIResponse(code=404, detail="文件已过期")
|
|
return (
|
|
APIResponse(detail=file_code.text)
|
|
if file_code.text
|
|
else response_from_download(await get_stored_download(file_code, file_storage))
|
|
)
|
|
|
|
|
|
chunk_api = APIRouter(prefix="/chunk", tags=["切片"])
|
|
|
|
|
|
async def parse_body_model(request: Request, model_class: type[BaseModel]):
|
|
content_type = request.headers.get("content-type", "").lower()
|
|
try:
|
|
if "application/x-www-form-urlencoded" in content_type or "multipart/form-data" in content_type:
|
|
payload = dict(await request.form())
|
|
else:
|
|
payload = await request.json()
|
|
return model_class.model_validate(payload)
|
|
except ValidationError as e:
|
|
raise HTTPException(status_code=422, detail=e.errors())
|
|
except Exception:
|
|
raise HTTPException(status_code=400, detail="请求体格式错误")
|
|
|
|
|
|
async def parse_init_chunk_upload(request: Request) -> InitChunkUploadModel:
|
|
return await parse_body_model(request, InitChunkUploadModel)
|
|
|
|
|
|
async def parse_complete_upload(request: Request) -> CompleteUploadModel:
|
|
return await parse_body_model(request, CompleteUploadModel)
|
|
|
|
|
|
@chunk_api.post("/upload/init/", dependencies=[Depends(share_required_login)])
|
|
async def init_chunk_upload(data: InitChunkUploadModel = Depends(parse_init_chunk_upload)):
|
|
safe_file_name = await sanitize_filename(unquote(data.file_name or ""))
|
|
validate_file_type(safe_file_name)
|
|
# 服务端校验:根据 total_chunks * chunk_size 计算理论最大上传量
|
|
total_chunks = (data.file_size + data.chunk_size - 1) // data.chunk_size
|
|
max_possible_size = total_chunks * data.chunk_size
|
|
if max_possible_size > settings.upload_size:
|
|
max_size_mb = settings.upload_size / (1024 * 1024)
|
|
raise HTTPException(
|
|
status_code=403, detail=f"文件大小超过限制,最大为 {max_size_mb:.2f} MB"
|
|
)
|
|
|
|
# 断点续传:检查是否存在相同文件的未完成上传会话
|
|
existing_session = await UploadChunk.filter(
|
|
chunk_hash=data.file_hash,
|
|
chunk_index=-1,
|
|
file_size=data.file_size,
|
|
file_name=safe_file_name,
|
|
).first()
|
|
|
|
if existing_session:
|
|
if not existing_session.save_path:
|
|
await UploadChunk.filter(upload_id=existing_session.upload_id).delete()
|
|
await release_storage(f"chunk:{existing_session.upload_id}")
|
|
else:
|
|
await reserve_storage(
|
|
f"chunk:{existing_session.upload_id}",
|
|
existing_session.file_size,
|
|
ttl_seconds=max(
|
|
1, int(getattr(settings, "chunk_expire_hours", 24))
|
|
)
|
|
* 3600,
|
|
)
|
|
uploaded_chunks = await UploadChunk.filter(
|
|
upload_id=existing_session.upload_id, completed=True
|
|
).values_list("chunk_index", flat=True)
|
|
return APIResponse(
|
|
detail={
|
|
"existed": False,
|
|
"upload_id": existing_session.upload_id,
|
|
"chunk_size": existing_session.chunk_size,
|
|
"total_chunks": existing_session.total_chunks,
|
|
"uploaded_chunks": list(uploaded_chunks),
|
|
}
|
|
)
|
|
|
|
# 创建新的上传会话
|
|
upload_id = uuid.uuid4().hex
|
|
reservation_token = f"chunk:{upload_id}"
|
|
chunk_expire_seconds = max(1, int(getattr(settings, "chunk_expire_hours", 24))) * 3600
|
|
await reserve_storage(
|
|
reservation_token, data.file_size, ttl_seconds=chunk_expire_seconds
|
|
)
|
|
_, _, _, safe_file_name, save_path = await get_chunk_file_path_name(
|
|
data.file_name, upload_id
|
|
)
|
|
try:
|
|
await UploadChunk.create(
|
|
upload_id=upload_id,
|
|
chunk_index=-1,
|
|
total_chunks=total_chunks,
|
|
file_size=data.file_size,
|
|
chunk_size=data.chunk_size,
|
|
chunk_hash=data.file_hash,
|
|
file_name=safe_file_name,
|
|
save_path=save_path,
|
|
)
|
|
except Exception:
|
|
await release_storage(reservation_token)
|
|
raise
|
|
return APIResponse(
|
|
detail={
|
|
"existed": False,
|
|
"upload_id": upload_id,
|
|
"chunk_size": data.chunk_size,
|
|
"total_chunks": total_chunks,
|
|
"uploaded_chunks": [],
|
|
}
|
|
)
|
|
|
|
|
|
@chunk_api.post(
|
|
"/upload/chunk/{upload_id}/{chunk_index}",
|
|
dependencies=[Depends(share_required_login)],
|
|
)
|
|
async def upload_chunk(
|
|
upload_id: str,
|
|
chunk_index: int,
|
|
chunk: UploadFile = File(...),
|
|
):
|
|
# 获取上传会话信息
|
|
chunk_info = await UploadChunk.filter(upload_id=upload_id, chunk_index=-1).first()
|
|
if not chunk_info:
|
|
raise HTTPException(status.HTTP_404_NOT_FOUND, detail="上传会话不存在")
|
|
await reserve_storage(
|
|
f"chunk:{upload_id}",
|
|
chunk_info.file_size,
|
|
ttl_seconds=max(1, int(getattr(settings, "chunk_expire_hours", 24))) * 3600,
|
|
)
|
|
|
|
# 检查分片索引有效性
|
|
if chunk_index < 0 or chunk_index >= chunk_info.total_chunks:
|
|
raise HTTPException(status.HTTP_400_BAD_REQUEST, detail="无效的分片索引")
|
|
|
|
# 检查是否已上传(支持断点续传)
|
|
existing_chunk = await UploadChunk.filter(
|
|
upload_id=upload_id, chunk_index=chunk_index, completed=True
|
|
).first()
|
|
if existing_chunk:
|
|
return APIResponse(
|
|
detail={"chunk_hash": existing_chunk.chunk_hash, "skipped": True}
|
|
)
|
|
|
|
# 读取分片数据并计算哈希
|
|
chunk_data = await chunk.read()
|
|
if chunk_index == 0:
|
|
validate_header_bytes(chunk_info.file_name, None, chunk_data[:64])
|
|
chunk_size = len(chunk_data)
|
|
|
|
# 校验分片大小不超过声明的 chunk_size
|
|
if chunk_size > chunk_info.chunk_size:
|
|
raise HTTPException(
|
|
status.HTTP_400_BAD_REQUEST,
|
|
detail=f"分片大小超过声明值: 最大 {chunk_info.chunk_size}, 实际 {chunk_size}",
|
|
)
|
|
|
|
# 计算已上传分片数,校验累计大小不超限(用分片数 * chunk_size 估算)
|
|
uploaded_count = await UploadChunk.filter(
|
|
upload_id=upload_id, completed=True
|
|
).count()
|
|
# 已上传分片的最大可能大小 + 当前分片
|
|
max_uploaded_size = uploaded_count * chunk_info.chunk_size + chunk_size
|
|
if max_uploaded_size > settings.upload_size:
|
|
max_size_mb = settings.upload_size / (1024 * 1024)
|
|
raise HTTPException(
|
|
status_code=403, detail=f"累计上传大小超过限制,最大为 {max_size_mb:.2f} MB"
|
|
)
|
|
|
|
chunk_hash = hashlib.sha256(chunk_data).hexdigest()
|
|
|
|
save_path = chunk_info.save_path
|
|
|
|
# 保存分片到存储
|
|
storage = storages[settings.file_storage]()
|
|
try:
|
|
await storage.save_chunk(
|
|
upload_id, chunk_index, chunk_data, chunk_hash, save_path
|
|
)
|
|
except Exception as e:
|
|
raise HTTPException(
|
|
status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"分片保存失败: {str(e)}"
|
|
)
|
|
|
|
# 更新或创建分片记录(保存成功后再记录)
|
|
await UploadChunk.update_or_create(
|
|
upload_id=upload_id,
|
|
chunk_index=chunk_index,
|
|
defaults={
|
|
"chunk_hash": chunk_hash,
|
|
"completed": True,
|
|
"file_size": chunk_info.file_size,
|
|
"total_chunks": chunk_info.total_chunks,
|
|
"chunk_size": chunk_info.chunk_size,
|
|
"file_name": chunk_info.file_name,
|
|
"save_path": chunk_info.save_path,
|
|
},
|
|
)
|
|
return APIResponse(detail={"chunk_hash": chunk_hash})
|
|
|
|
|
|
@chunk_api.delete("/upload/{upload_id}", dependencies=[Depends(share_required_login)])
|
|
async def cancel_upload(upload_id: str):
|
|
"""取消上传并清理临时文件"""
|
|
chunk_info = await UploadChunk.filter(upload_id=upload_id, chunk_index=-1).first()
|
|
if not chunk_info:
|
|
raise HTTPException(status.HTTP_404_NOT_FOUND, detail="上传会话不存在")
|
|
|
|
save_path = chunk_info.save_path
|
|
|
|
# 清理存储中的临时文件
|
|
storage = storages[settings.file_storage]()
|
|
if save_path:
|
|
try:
|
|
await storage.clean_chunks(upload_id, save_path)
|
|
except Exception:
|
|
logger.warning("取消分片上传:清理分片文件失败 upload_id=%s", upload_id, exc_info=True)
|
|
|
|
# 清理数据库记录
|
|
await UploadChunk.filter(upload_id=upload_id).delete()
|
|
await release_storage(f"chunk:{upload_id}")
|
|
|
|
return APIResponse(detail={"message": "上传已取消"})
|
|
|
|
|
|
@chunk_api.get(
|
|
"/upload/status/{upload_id}", dependencies=[Depends(share_required_login)]
|
|
)
|
|
async def get_upload_status(upload_id: str):
|
|
"""获取上传状态"""
|
|
chunk_info = await UploadChunk.filter(upload_id=upload_id, chunk_index=-1).first()
|
|
if not chunk_info:
|
|
raise HTTPException(status.HTTP_404_NOT_FOUND, detail="上传会话不存在")
|
|
|
|
# 获取已上传的分片列表
|
|
uploaded_chunks = await UploadChunk.filter(
|
|
upload_id=upload_id, completed=True
|
|
).values_list("chunk_index", flat=True)
|
|
|
|
return APIResponse(
|
|
detail={
|
|
"upload_id": upload_id,
|
|
"file_name": chunk_info.file_name,
|
|
"file_size": chunk_info.file_size,
|
|
"chunk_size": chunk_info.chunk_size,
|
|
"total_chunks": chunk_info.total_chunks,
|
|
"uploaded_chunks": list(uploaded_chunks),
|
|
"progress": len(uploaded_chunks) / chunk_info.total_chunks * 100,
|
|
}
|
|
)
|
|
|
|
|
|
@chunk_api.post(
|
|
"/upload/complete/{upload_id}", dependencies=[Depends(share_required_login)]
|
|
)
|
|
async def complete_upload(
|
|
upload_id: str,
|
|
data: CompleteUploadModel = Depends(parse_complete_upload),
|
|
ip: str = Depends(ip_limit["upload"]),
|
|
):
|
|
# 获取上传基本信息
|
|
chunk_info = await UploadChunk.filter(upload_id=upload_id, chunk_index=-1).first()
|
|
if not chunk_info:
|
|
raise HTTPException(status.HTTP_404_NOT_FOUND, detail="上传会话不存在")
|
|
validate_expire_style(data.expire_style)
|
|
detail = await FileUploadService.complete_chunked_upload(
|
|
upload_id, chunk_info, data.expire_value, data.expire_style
|
|
)
|
|
ip_limit["upload"].add_ip(ip)
|
|
return APIResponse(detail=detail)
|
|
|
|
|
|
# ============ 预签名上传API ============
|
|
presign_api = APIRouter(prefix="/presign", tags=["预签名上传"])
|
|
|
|
|
|
def build_proxy_upload_urls(upload_id: str) -> dict:
|
|
proxy_upload_url = f"/presign/upload/proxy/{upload_id}"
|
|
return {
|
|
"proxy_upload_url": proxy_upload_url,
|
|
"legacy_proxy_upload_url": f"/api{proxy_upload_url}",
|
|
}
|
|
|
|
|
|
async def _get_valid_session(
|
|
upload_id: str, expected_mode: Optional[str] = None
|
|
) -> PresignUploadSession:
|
|
"""获取并验证会话"""
|
|
session = await PresignUploadSession.filter(upload_id=upload_id).first()
|
|
if not session:
|
|
raise HTTPException(404, "上传会话不存在")
|
|
if await session.is_expired():
|
|
await session.delete()
|
|
await release_storage(f"presign:{upload_id}")
|
|
raise HTTPException(404, "上传会话已过期")
|
|
if expected_mode and session.mode != expected_mode:
|
|
raise HTTPException(400, f"此会话不支持{expected_mode}模式")
|
|
return session
|
|
|
|
|
|
@presign_api.post("/upload/init", dependencies=[Depends(share_required_login)])
|
|
async def presign_upload_init(
|
|
data: PresignUploadInitRequest, ip: str = Depends(ip_limit["upload"])
|
|
):
|
|
"""初始化预签名上传,S3返回直传URL,其他存储返回代理URL"""
|
|
validate_file_type(data.file_name)
|
|
if data.file_size > settings.upload_size:
|
|
raise HTTPException(
|
|
403,
|
|
f"文件大小超过限制,最大为 {settings.upload_size / (1024 * 1024):.2f} MB",
|
|
)
|
|
validate_expire_style(data.expire_style)
|
|
|
|
upload_id = uuid.uuid4().hex
|
|
reservation_token = f"presign:{upload_id}"
|
|
await reserve_storage(
|
|
reservation_token, data.file_size, ttl_seconds=PRESIGN_SESSION_EXPIRES
|
|
)
|
|
try:
|
|
path, _, _, filename, save_path = await FileUploadService.generate_file_path(
|
|
data.file_name, upload_id
|
|
)
|
|
storage: FileStorageInterface = storages[settings.file_storage]()
|
|
presigned_url = await storage.generate_presigned_upload_url(
|
|
save_path, PRESIGN_SESSION_EXPIRES
|
|
)
|
|
mode = "direct" if presigned_url else "proxy"
|
|
proxy_urls = build_proxy_upload_urls(upload_id)
|
|
upload_url = presigned_url or proxy_urls["proxy_upload_url"]
|
|
await PresignUploadSession.create(
|
|
upload_id=upload_id,
|
|
file_name=filename,
|
|
file_size=data.file_size,
|
|
save_path=save_path,
|
|
mode=mode,
|
|
expire_value=data.expire_value,
|
|
expire_style=data.expire_style,
|
|
expires_at=await get_now() + timedelta(seconds=PRESIGN_SESSION_EXPIRES),
|
|
)
|
|
except Exception:
|
|
await release_storage(reservation_token)
|
|
raise
|
|
|
|
ip_limit["upload"].add_ip(ip)
|
|
detail = {
|
|
"upload_id": upload_id,
|
|
"upload_url": upload_url,
|
|
"mode": mode,
|
|
"expires_in": PRESIGN_SESSION_EXPIRES,
|
|
}
|
|
if mode == "proxy":
|
|
detail.update(proxy_urls)
|
|
|
|
return APIResponse(
|
|
detail=detail
|
|
)
|
|
|
|
|
|
@presign_api.put(
|
|
"/upload/proxy/{upload_id}", dependencies=[Depends(share_required_login)]
|
|
)
|
|
async def presign_upload_proxy(
|
|
upload_id: str, file: UploadFile = File(...), ip: str = Depends(ip_limit["upload"])
|
|
):
|
|
"""代理模式上传,服务器转存到存储后端"""
|
|
session = await _get_valid_session(upload_id, expected_mode="proxy")
|
|
code = await FileUploadService.commit_proxy_upload(session, file)
|
|
ip_limit["upload"].add_ip(ip)
|
|
return APIResponse(detail={"code": code, "name": session.file_name})
|
|
|
|
|
|
@presign_api.post(
|
|
"/upload/confirm/{upload_id}", dependencies=[Depends(share_required_login)]
|
|
)
|
|
async def presign_upload_confirm(upload_id: str, ip: str = Depends(ip_limit["upload"])):
|
|
"""直传确认,客户端完成S3直传后调用获取分享码"""
|
|
session = await _get_valid_session(upload_id, expected_mode="direct")
|
|
code = await FileUploadService.confirm_direct_upload(session)
|
|
ip_limit["upload"].add_ip(ip)
|
|
return APIResponse(detail={"code": code, "name": session.file_name})
|
|
|
|
|
|
@presign_api.get(
|
|
"/upload/status/{upload_id}", dependencies=[Depends(share_required_login)]
|
|
)
|
|
async def presign_upload_status(upload_id: str):
|
|
"""查询上传会话状态"""
|
|
session = await PresignUploadSession.filter(upload_id=upload_id).first()
|
|
if not session:
|
|
raise HTTPException(404, "上传会话不存在")
|
|
|
|
return APIResponse(
|
|
detail={
|
|
"upload_id": session.upload_id,
|
|
"file_name": session.file_name,
|
|
"file_size": session.file_size,
|
|
"mode": session.mode,
|
|
"created_at": session.created_at.isoformat(),
|
|
"expires_at": session.expires_at.isoformat(),
|
|
"is_expired": await session.is_expired(),
|
|
}
|
|
)
|
|
|
|
|
|
@presign_api.delete("/upload/{upload_id}", dependencies=[Depends(share_required_login)])
|
|
async def presign_upload_cancel(upload_id: str):
|
|
"""取消上传会话"""
|
|
session = await PresignUploadSession.filter(upload_id=upload_id).first()
|
|
if not session:
|
|
raise HTTPException(404, "上传会话不存在")
|
|
|
|
if session.mode == "direct":
|
|
storage: FileStorageInterface = storages[settings.file_storage]()
|
|
try:
|
|
if await storage.file_exists(session.save_path):
|
|
temp_file_code = StoredFile(
|
|
file_path=os.path.dirname(session.save_path),
|
|
uuid_file_name=os.path.basename(session.save_path),
|
|
)
|
|
await storage.delete_file(temp_file_code)
|
|
except Exception:
|
|
logger.warning("取消预签名会话:清理临时文件失败 upload_id=%s", upload_id, exc_info=True)
|
|
|
|
await session.delete()
|
|
await release_storage(f"presign:{upload_id}")
|
|
return APIResponse(detail={"message": "上传会话已取消"})
|