mirror of
https://github.com/vastsa/FileCodeBox.git
synced 2026-11-20 13:29:56 +08:00
* fix: constant-time comparison for download tokens (security item 5) Replace the set-membership key check in /share/download with hmac.compare_digest over the two valid window tokens; add negative-path tests (wrong key 403, both windows accepted, foreign-code token 403). * fix: single source for attachment headers + guard tests (security item 2) All six Content-Disposition construction sites across the five storage backends now go through build_attachment_headers; the header is what neutralizes stored XSS on the same-origin download path. Guard tests assert every get_file_response uses the builder and no hand-built disposition reappears. * fix: reject internal-network endpoints for storage config (security item 7) s3_endpoint_url/s3_hostname/webdav_url are fetched server-side; with APP_ENV=production the write entry now enforces http(s) schemes and a loopback/private/link-local/metadata host blacklist. Development env stays open for local minio/webdav. Already-stored values are never re-validated, so existing deployments are unaffected. * fix: JSON 404 for handler-raised 404s; negative-path test batch Starlette status-code handlers take precedence over the HTTPException class handler, so registering the theme index as the 404 handler turned every HTTPException(404) app-wide into a 200 HTML page (found by the new negative-path tests). The new not_found_handler serves the theme page only to browsers (Accept: text/html) and JSON 404 to API clients. Negative-path batch: download count exhaustion, missing chunk session 404s, expired presign session deletion, admin update_file uniqueness/ existence. * fix: run container as non-root app user with compat volume chown (security item 3) Entrypoint starts as root only to fix the data-dir ownership (skipped when nothing needs changing or when the deployer sets an explicit user), then drops to uid 10001 via gosu. Verified in-container: uvicorn runs as app (uid 10001), setup and share round-trip work against a root-owned volume. * fix: compare download tokens as bytes (non-ASCII key caused 500) * test: lock changed-only semantics for endpoint config saves Integration regression for the background lesson: a stored internal endpoint must not block unrelated settings saves in production mode; changing to an internal endpoint (URL or bare-hostname form) is 400. * fix: resolve-based SSRF check — deny IP shorthands, hex/decimal IPs, DNS maps Probe found the static blacklist bypassed by glibc shorthands (127.1, 10.1), decimal (2130706433) and hex IP forms, *.localhost, and loopback-mapping DNS services. Both validators now resolve the host and re-check every resolved address; hostname tier extracts the host part correctly for [v6]/host:port/bare-v6 forms. 44 tests. * test: pin 404 handler dual-branch behavior (browser theme page vs API JSON) * fix: write back normalized endpoint values; hostname dev-gate test * test: header-injection negative paths for attachment disposition * fix: S3 merge buffers parts to >=5MiB; missing object 404 upfront - S3 multipart rejects parts <5MiB (EntityTooSmall) except the last; chunk size is client-controlled (commonly 2-4MB) so multi-chunk merges always failed with 500. Merge now buffers chunks into >=5MiB parts (memory bound: 5MB + one chunk). - get_file_response: a 404 head_object now raises StorageError(404) upfront instead of signing a doomed presigned URL and failing mid-stream (aligns with local backend semantics from the M2 behavior unification). * test: S3 backend coverage via in-process moto server (13 cases) mock_aws cannot intercept aioboto3's aiohttp stack, so tests run against a real in-process ThreadedMotoServer endpoint. Covers roundtrips, missing object 404, merge failure modes (missing chunk / hash mismatch -> abort, no leftover object), cleanup scoping, presign shapes, proxy dispatch. Also adds moto[s3,server] to the dev group and CI. * fix: WebDAV missing object 404 upfront; connection errors map to 503 HEAD 404 was silently swallowed (signed a 200 bad stream that failed mid-download, same family as the S3 bug), and HEAD connection errors were also swallowed so the 503 mapping never fired. Both now surface properly; the aiohttp session is reclaimed on pre-stream exception paths. * test: WebDAV backend coverage via in-process WebDAV server (9 cases) Real HTTP (HEAD/GET/PUT/DELETE/MKCOL/PROPFIND) against a minimal aiohttp WebDAV server over a temp dir: roundtrips, missing object 404, connection error 503, delete with empty-parent cleanup, merge failure modes, chunk cleanup scoping. * fix: OneDrive missing object maps to 404 upfront; OpenDAL behavior pinned OneDrive get_file_response translated graph itemNotFound into the outer 503 catch-all; now raises StorageError(404) like local/S3/WebDAV. OpenDAL already mapped missing objects to 404 via its outer catch — pinned with fake-operator tests (SDK not in runtime deps, instances built via __new__). Also tightens the S3 presign URL assertion. * test: admin write-path coverage (batch/single delete, batch update, policy actions) 13 negative-path cases for the previously untested data-modifying admin endpoints: mixed-id aggregation with duplicates and missing records, empty-list rejections, clear_expired_at permanence semantics, policy action boundaries (zero limit 400, unknown action 400, missing 404), and the DoesNotExist->404 mapping pinned for single delete. * test: admin read-path and view-preset CRUD coverage (22 cases) Detail/metadata/preview/admin-download/activities/local-lists+delete/ verify: missing-record 404s, note/tag truncation limits, preview max_chars truncation, activity filtering and the 80-event clamp, local path traversal rejection. Presets: update-vs-create by id, name truncation, filter normalization clamps, 24-preset cap, delete-missing 404. * refactor: move auth primitives to apps/base/auth, kill base->admin dep JWT create/verify, bearer extraction and the share-upload gate move to apps.base.auth (consumed by both surfaces); admin.dependencies keeps admin session gating and service providers, re-exporting the primitives for import-path compatibility. base.views no longer imports from apps.admin — the last cross-app reverse dependency is gone. * refactor: split admin services god-module (ConfigService/LocalFile out) apps/admin/services.py (1697 lines) becomes FileService (~1200 lines) + a compatibility facade re-exporting ConfigService, LocalFileService, LocalFileClass and keyvalue_write_lock from their new homes (config_service.py holds the D5 KeyValue lock; one-way imports, no cycle). test_admin_security patch targets follow ConfigService to its new module. Behavior unchanged; 247 tests green. * fix: size-less uploads no longer crash (upstream seek bug) validate_file_size's size-None branch called UploadFile.seek(0, 2), which raises TypeError (UploadFile.seek takes one arg); the underlying SpooledTemporaryFile.seek is sync, so awaiting it also fails. Use the underlying file object with sync seek(0, SEEK_END). Surface discovered by the mypy pilot run; regression test covers the size-None branch. * chore: bump direct deps (patch/minor) fastapi 0.139.2->0.141.1, pydantic 2.12.5->2.13.5, uvicorn 0.51.0->0.53.0, aiohttp 3.14.2->3.14.3; lockfile regenerated. tortoise-orm 0.x->1.x major deliberately deferred pending API-change review. * ci: gradual mypy adoption via baseline ratchet scripts/mypy_ratchet.py compares mypy output against scripts/mypy-baseline.txt (line numbers stripped for edit stability): new type errors fail CI, fixes are folded back via --regenerate. Baseline starts at 30 known errors; the seek bug found in the pilot was fixed before baselining. mypy joins the dev group and CI. * fix: migrate WebDAV auth off aiohttp.BasicAuth (deprecated in 4.0) Dependency-upgrade probe on aiohttp 3.14.3 surfaced the BasicAuth deprecation; switch to aiohttp.encode_basic_auth() headers so the aiohttp 4.0 upgrade does not break the WebDAV backend. * chore: drop dead code (WebDAV _instance stub, unused LocalFileClass.write) * test: file-validation negative paths (13 cases) Magic-bytes spoofing (text-as-png, exe-as-pdf, mp4/webp box detection), whitelist rule semantics (star/image-wildcard/content mismatch), chunk-0 header validation parity, and end-to-end UploadFile rejection. * test: migration runner coverage (5 cases) Full chain executes 001-007 in filename order and registers; rerun is idempotent; pre-registered entries are skipped (DDL not re-executed); the resulting schema matches deployment; a failing migration propagates and is not registered. Uses a fresh :memory: DB without generate_schemas so the migrations themselves build the schema (the real deployment path). * refactor: split core/storage.py into a package (per-backend modules) core/storage.py (1400+ lines, six backends) becomes core/storage/ with _base.py (data contracts, interface, shared header builder), and local/s3/onedrive/opendal/webdav modules. The package __init__ re-exports the full historical public surface; test patch targets move to the precise backend modules (core.storage.local.data_root etc.) and the attachment guard scans the package. Import-only change; 264 tests green. Storage-layer DI deferred: current tests construct instances against patched settings, so the added surface isn't justified yet. * refactor: TypedDict for IPRateLimit records The rate-limit ledger mixed int and datetime under a loose Union typing, producing 5 mypy noise errors. A TypedDict gives precise per-key types; setdefault replaces the get-then-assign pattern. One mypy baseline error family resolved; 264 tests green. * chore: tighten mypy baseline after IPRateLimit TypedDict * refactor: resolve all 26 baseline mypy errors; precise test assertions Per-error treatment (no bulk script): honest annotations for StoredDownload (Path/bytes/Callable), KeyValue.value (Any for JSONField), dict[str, Any] for form/config dicts; signature split in create_token; str() normalization in config int(); platform/type-ignore only where the checker cannot follow (msvcrt, FastAPI Request injection). Baseline file and ratchet compare now redundant at zero errors but kept as the guard against regressions. Tests: assertions tightened to status_code == 403 with HTTPException raises; _create_share explicit parameters replace **extra passthrough; seek-fix pointer-reset assertion persisted.
355 lines
15 KiB
Python
355 lines
15 KiB
Python
import base64
|
||
import hashlib
|
||
import os
|
||
import tempfile
|
||
from core.logger import logger
|
||
from typing import BinaryIO, Optional
|
||
from urllib.parse import quote, unquote
|
||
|
||
import aiofiles
|
||
import aiohttp
|
||
import asyncio
|
||
from pathlib import Path
|
||
import re
|
||
from core.errors import StorageError
|
||
from core.settings import settings
|
||
from core.utils import get_file_url, sanitize_filename
|
||
from starlette.background import BackgroundTask
|
||
|
||
from core.storage._base import (
|
||
FileStorageInterface,
|
||
StoredDownload,
|
||
StoredFile,
|
||
build_attachment_headers,
|
||
)
|
||
|
||
|
||
class WebDAVFileStorage(FileStorageInterface):
|
||
def __init__(self):
|
||
if not hasattr(self, "_initialized"):
|
||
self.base_url = settings.webdav_url.rstrip("/") + "/"
|
||
# aiohttp 4.0 移除 BasicAuth(auth=...) 参数——改用编码后的 Authorization 头
|
||
self.auth_headers = {
|
||
"Authorization": aiohttp.encode_basic_auth(
|
||
settings.webdav_username, settings.webdav_password
|
||
)
|
||
}
|
||
self._initialized = True
|
||
|
||
def _build_url(self, path: str) -> str:
|
||
encoded_path = quote(str(path.replace("\\", "/").lstrip("/")).lstrip("/"))
|
||
return f"{self.base_url}{encoded_path}"
|
||
|
||
async def _mkdir_p(self, directory_path: str):
|
||
"""递归创建目录(类似mkdir -p)"""
|
||
path_obj = Path(unquote(directory_path))
|
||
current_path = ""
|
||
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
# 逐级检查目录是否存在
|
||
for part in path_obj.parts:
|
||
current_path = str(Path(current_path) / part)
|
||
url = self._build_url(current_path)
|
||
|
||
# 检查目录是否存在
|
||
async with session.head(url) as resp:
|
||
if resp.status == 404:
|
||
# 创建目录
|
||
async with session.request("MKCOL", url) as mkcol_resp:
|
||
if mkcol_resp.status not in (200, 201, 409):
|
||
content = await mkcol_resp.text()
|
||
raise StorageError(
|
||
status_code=mkcol_resp.status,
|
||
detail=f"目录创建失败: {content[:200]}",
|
||
)
|
||
|
||
async def _is_dir_empty(self, dir_path: str) -> bool:
|
||
"""检查目录是否为空"""
|
||
url = self._build_url(dir_path)
|
||
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
async with session.request("PROPFIND", url, headers={"Depth": "1"}) as resp:
|
||
if resp.status != 207: # 207 是 Multi-Status 响应
|
||
return False
|
||
content = await resp.text()
|
||
# 如果只有一个 response(当前目录),说明目录为空
|
||
return content.count("<D:response>") <= 1
|
||
|
||
async def _delete_empty_dirs(self, file_path: str, session: aiohttp.ClientSession):
|
||
"""递归删除空目录"""
|
||
path_obj = Path(file_path)
|
||
current_path = path_obj.parent
|
||
|
||
while str(current_path) != ".":
|
||
if not await self._is_dir_empty(str(current_path)):
|
||
break
|
||
|
||
url = self._build_url(str(current_path))
|
||
async with session.delete(url) as resp:
|
||
if resp.status not in (200, 204, 404):
|
||
break
|
||
|
||
current_path = current_path.parent
|
||
|
||
async def save_file(
|
||
self, stream: BinaryIO, save_path: str, content_type: Optional[str] = None
|
||
):
|
||
"""保存文件(自动创建目录,流式上传)"""
|
||
path_obj = Path(save_path)
|
||
directory_path = str(path_obj.parent)
|
||
# 提取原始文件名并进行清理
|
||
filename = await sanitize_filename(path_obj.name)
|
||
# 构建安全的保存路径
|
||
safe_save_path = str(Path(directory_path) / filename)
|
||
|
||
try:
|
||
# 先创建目录结构
|
||
await self._mkdir_p(directory_path)
|
||
# 上传文件(流式)
|
||
url = self._build_url(safe_save_path)
|
||
|
||
async def file_sender():
|
||
"""流式读取文件内容"""
|
||
chunk_size = 256 * 1024 # 256KB chunks
|
||
while True:
|
||
chunk = await asyncio.to_thread(stream.read, chunk_size)
|
||
if not chunk:
|
||
break
|
||
yield chunk
|
||
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
async with session.put(
|
||
url,
|
||
data=file_sender(),
|
||
headers={"Content-Type": content_type or "application/octet-stream"}
|
||
) as resp:
|
||
if resp.status not in (200, 201, 204):
|
||
content = await resp.text()
|
||
raise StorageError(
|
||
status_code=resp.status,
|
||
detail=f"文件上传失败: {content[:200]}",
|
||
)
|
||
except aiohttp.ClientError as e:
|
||
raise StorageError(
|
||
status_code=503, detail=f"WebDAV连接异常: {str(e)}")
|
||
|
||
async def delete_file(self, file_code: StoredFile):
|
||
"""删除WebDAV文件及空目录"""
|
||
file_path = file_code.get_file_path()
|
||
url = self._build_url(file_path)
|
||
try:
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
# 删除文件
|
||
async with session.delete(url) as resp:
|
||
if resp.status not in (200, 204, 404):
|
||
content = await resp.text()
|
||
raise StorageError(
|
||
status_code=resp.status,
|
||
detail=f"WebDAV删除失败: {content[:200]}",
|
||
)
|
||
|
||
# 使用同一个 session 删除空目录
|
||
await self._delete_empty_dirs(file_path, session)
|
||
|
||
except aiohttp.ClientError as e:
|
||
raise StorageError(
|
||
status_code=503, detail=f"WebDAV连接异常: {str(e)}")
|
||
|
||
async def get_file_url(self, file_code: StoredFile):
|
||
return await get_file_url(file_code.code)
|
||
|
||
async def get_file_response(self, file_code: StoredFile):
|
||
"""获取文件响应(代理模式)"""
|
||
try:
|
||
filename = file_code.prefix + file_code.suffix
|
||
url = self._build_url(file_code.get_file_path())
|
||
content_length = None # 初始化为 None,表示未知大小
|
||
|
||
# 创建ClientSession并复用(包含认证头)
|
||
session = aiohttp.ClientSession(headers={
|
||
"Authorization": f"Basic {base64.b64encode(f'{settings.webdav_username}:{settings.webdav_password}'.encode()).decode()}"
|
||
})
|
||
|
||
# 尝试发送HEAD请求获取Content-Length;对象不存在时前置 404
|
||
# (与 local/S3 语义对齐),连接层错误映射 503——两者都不能
|
||
# 静默吞掉后签出 200 坏流。异常路径回收 session 防泄漏。
|
||
try:
|
||
try:
|
||
async with session.head(url) as resp:
|
||
if resp.status == 404:
|
||
raise StorageError(
|
||
status_code=404, detail="文件已过期删除"
|
||
)
|
||
if resp.status == 200 and 'Content-Length' in resp.headers:
|
||
content_length = int(resp.headers['Content-Length'])
|
||
except StorageError:
|
||
raise
|
||
except aiohttp.ClientError as e:
|
||
raise StorageError(
|
||
status_code=503, detail=f"WebDAV连接异常: {str(e)}"
|
||
) from e
|
||
except Exception:
|
||
# 其他 HEAD 异常不阻断:流式下载阶段会给出真实状态
|
||
pass
|
||
|
||
except BaseException:
|
||
await session.close()
|
||
raise
|
||
|
||
async def stream_generator():
|
||
try:
|
||
async with session.get(url) as resp:
|
||
if resp.status != 200:
|
||
raise StorageError(
|
||
status_code=resp.status,
|
||
detail=f"文件获取失败{resp.status}: {await resp.text()}",
|
||
)
|
||
chunk_size = 65536
|
||
while True:
|
||
chunk = await resp.content.read(chunk_size)
|
||
if not chunk:
|
||
break
|
||
yield chunk
|
||
finally:
|
||
await session.close()
|
||
|
||
headers = build_attachment_headers(filename, content_length)
|
||
return StoredDownload(
|
||
filename=filename,
|
||
headers=headers,
|
||
stream_factory=stream_generator,
|
||
# 兜底关闭会话:客户端中断时与 generator finally 双保险
|
||
background=BackgroundTask(session.close),
|
||
)
|
||
except aiohttp.ClientError as e:
|
||
raise StorageError(
|
||
status_code=503, detail=f"WebDAV连接异常: {str(e)}")
|
||
|
||
async def save_chunk(self, upload_id: str, chunk_index: int, chunk_data: bytes, chunk_hash: str, save_path: str):
|
||
"""保存分片到 WebDAV"""
|
||
chunk_dir = str(Path(save_path).parent / "chunks" / upload_id)
|
||
chunk_path = f"{chunk_dir}/{chunk_index}.part"
|
||
|
||
# 先创建目录结构
|
||
await self._mkdir_p(chunk_dir)
|
||
|
||
chunk_url = self._build_url(chunk_path)
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
async with session.put(chunk_url, data=chunk_data) as resp:
|
||
if resp.status not in (200, 201, 204):
|
||
content = await resp.text()
|
||
raise StorageError(
|
||
status_code=resp.status,
|
||
detail=f"分片上传失败: {content[:200]}"
|
||
)
|
||
|
||
async def merge_chunks(self, upload_id: str, total_chunks: int, chunk_size: int, save_path: str, chunk_records: dict) -> tuple[str, str]:
|
||
"""
|
||
合并 WebDAV 上的分片文件
|
||
使用临时文件避免内存问题
|
||
"""
|
||
file_sha256 = hashlib.sha256()
|
||
chunk_dir = str(Path(save_path).parent / "chunks" / upload_id)
|
||
|
||
# 使用临时文件存储合并数据,避免内存问题
|
||
with tempfile.NamedTemporaryFile(delete=False) as temp_file:
|
||
temp_path = temp_file.name
|
||
|
||
try:
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
# 按顺序读取并验证每个分片,写入临时文件
|
||
async with aiofiles.open(temp_path, 'wb') as out_file:
|
||
for i in range(total_chunks):
|
||
chunk_path = f"{chunk_dir}/{i}.part"
|
||
chunk_url = self._build_url(chunk_path)
|
||
|
||
# 获取分片记录
|
||
chunk_record = self._get_chunk_record(chunk_records, i)
|
||
|
||
# 下载分片数据
|
||
async with session.get(chunk_url) as resp:
|
||
if resp.status != 200:
|
||
raise ValueError(f"分片{i}文件不存在或无法访问")
|
||
chunk_data = await resp.read()
|
||
|
||
# 验证哈希
|
||
self._verify_and_hash_chunk(i, chunk_record, chunk_data, file_sha256)
|
||
await out_file.write(chunk_data)
|
||
del chunk_data # 释放内存
|
||
|
||
# 确保目标目录存在
|
||
output_dir = str(Path(save_path).parent)
|
||
await self._mkdir_p(output_dir)
|
||
|
||
# 流式上传合并后的文件
|
||
output_url = self._build_url(save_path)
|
||
|
||
async def file_sender():
|
||
async with aiofiles.open(temp_path, 'rb') as f:
|
||
while True:
|
||
chunk = await f.read(256 * 1024)
|
||
if not chunk:
|
||
break
|
||
yield chunk
|
||
|
||
async with session.put(output_url, data=file_sender()) as resp:
|
||
if resp.status not in (200, 201, 204):
|
||
content = await resp.text()
|
||
raise StorageError(
|
||
status_code=resp.status,
|
||
detail=f"合并文件上传失败: {content[:200]}"
|
||
)
|
||
finally:
|
||
# 清理临时文件
|
||
if os.path.exists(temp_path):
|
||
os.unlink(temp_path)
|
||
|
||
return save_path, file_sha256.hexdigest()
|
||
|
||
async def clean_chunks(self, upload_id: str, save_path: str):
|
||
"""
|
||
清理 WebDAV 上的临时分片文件
|
||
:param upload_id: 上传会话ID
|
||
:param save_path: 文件保存路径
|
||
"""
|
||
chunk_dir = str(Path(save_path).parent / "chunks" / upload_id)
|
||
chunk_dir_url = self._build_url(chunk_dir)
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
try:
|
||
# 检查分片目录是否存在
|
||
async with session.request("PROPFIND", chunk_dir_url, headers={"Depth": "1"}) as resp:
|
||
if resp.status == 207: # 207 表示 Multi-Status
|
||
# 获取目录下的所有分片文件
|
||
xml_data = await resp.text()
|
||
file_paths = re.findall(
|
||
r'<D:href>(.*?)</D:href>', xml_data)
|
||
for file_path in file_paths:
|
||
if file_path.endswith(".part"):
|
||
# 删除分片文件
|
||
file_url = self._build_url(file_path)
|
||
async with session.delete(file_url) as delete_resp:
|
||
if delete_resp.status not in (200, 204, 404):
|
||
logger.warning(f"删除分片文件失败: {file_path}")
|
||
|
||
# 删除分片目录
|
||
async with session.delete(chunk_dir_url) as delete_resp:
|
||
if delete_resp.status not in (200, 204, 404):
|
||
logger.warning(f"删除分片目录失败: {chunk_dir_url}")
|
||
else:
|
||
logger.info(f"分片目录不存在: {chunk_dir_url}")
|
||
except Exception as e:
|
||
logger.warning(f"清理 WebDAV 分片时出错: {e}")
|
||
|
||
async def file_exists(self, save_path: str) -> bool:
|
||
"""
|
||
检查文件是否存在于WebDAV
|
||
:param save_path: 文件路径
|
||
:return: 文件是否存在
|
||
"""
|
||
url = self._build_url(save_path)
|
||
async with aiohttp.ClientSession(headers=self.auth_headers) as session:
|
||
async with session.head(url) as resp:
|
||
return resp.status == 200
|
||
|
||
|