What

• Unstract's file-type gate now classifies files with the same steps as LLMWhisperer's (Util.is_valid_file_type_from_path in unstract-llm-whisperer), so a file LLMWhisperer would accept is no longer rejected here.
• API deployment uploads, Prompt Studio uploads and FileValidator classify the whole file. Connector/ETL copies (backend and workers) check the first chunk and classify the whole file locally only when that check is unclear.

Why

• A customer's ERP (SAP) xlsx exports were rejected with unsupported MIME type 'application/octet-stream', while LLMWhisperer extracted the same file fine (UN-4574). Re-saving in Excel "fixed" it.
• The export's first zip member is a leftover [trash]/0002.dat part. For that layout libmagic describes the file as "Microsoft OOXML" but reports MIME application/octet-stream, not application/zip.
• Our gate only looked inside the zip when libmagic said exactly application/zip, so the file stopped at octet-stream. LLMWhisperer asks Magika after an octet-stream or zip verdict, and Magika says xlsx.
• The connector paths and Prompt Studio only ever sniffed a prefix, so they rejected the same file.

How

• unstract/core/mime_gate.py:
  • classify_file_path(path) copies LLMWhisperer's gate step by step: magic.from_file on the whole file → for octet-stream/zip, a second look via our zip reader (identify_zip_container, standing in for Magika, which we deliberately don't add as a dependency) → for octet-stream/zip still, LLMWhisperer's %PDF- scan of the first 1024 bytes.
  • classify_stream(stream, chunk_size) is for connector copies. A first chunk libmagic can name decides on its own. An unclear one (octet-stream/zip/OLE2) is copied to a local temp file and classified whole. The copy is capped at LLMWhisperer's default MAX_FILE_SIZE (300 MB); larger files keep the first-chunk verdict and are rejected, as LLMWhisperer would reject them anyway. The bytes are handed back for the copy, so nothing is read twice from the source.
  • Removed resolve_inconclusive_mime_type, which was stricter than LLMWhisperer: it needed %PDF-x.y, re-checked with libmagic, and skipped zips.
• backend/utils/upload_mime_type.py: detect_upload_mime_type classifies an upload from its temp file, or copies it to one first. A temp dir that can't be written raises UploadSpillError (a RuntimeError), so it fails the request instead of being reported as a bad file.
• Callers switched over: the API staging loop and connector copy in endpoint_v2/source.py, the worker connector copy in workers/shared/workflow/execution/service.py, FileValidator (which now checks size before type), and Prompt Studio's upload view.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• More files are accepted, by design. This PR copies LLMWhisperer's %PDF- rule exactly: any file libmagic reports as octet-stream or zip with %PDF- in its first 1024 bytes is accepted as a PDF. That includes a zip with an uncompressed entry containing the marker. LLMWhisperer accepts these too, so they don't fail later in extraction.
• Known remaining gap: without Magika, LLMWhisperer can still name some octet-stream/zip files that we reject. It never goes the other way: we don't accept anything LLMWhisperer refuses.
• Local disk use: every upload under Django's in-memory limit (2.5 MB) is now copied to a temp file to be classified. Connector files whose first chunk is unclear are copied to the pod's /tmp, up to 300 MB. A legacy .doc/.xls or an unclear binary in an ETL source now costs a local copy where it used to be judged from its first 4 MB.
• Error order: FileValidator now reports an oversized file as too large before checking its type.
• Removed symbols (resolve_inconclusive_mime_type, and the INCONCLUSIVE_MIME_TYPES re-exports from workflow_manager.endpoint_v2.enums and shared.enums.file_types) have no users in unstract or unstract-cloud main.

Database Migrations

• None

Env Config

• None

Relevant Docs

• LLMWhisperer gate: unstract-llm-whisperer → backend/app/util/base.py Util.is_valid_file_type_from_path

Related Issues or PRs

• https://zipstack.atlassian.net/browse/UN-4574
• Follows up UN-1924 [FIX] Reject unsupported files in API deployment #2267 (UN-1924), which introduced the gate.
• Companion cloud PR (Prompt Studio's file_converter plugin): Zipstack/unstract-cloud#1941. It depends on this PR.

Dependencies Versions

• No new dependencies.

Notes on Testing

• Unit tests: backend/workflow_manager/endpoint_v2/tests/test_api_storage_mime_validation.py: 56 passed. These include an ERP-shaped xlsx (synthetic, built in the test) through the API staging loop and through the backend connector copy, and a full temp dir failing the request. workers/tests/test_mime_gate.py: 9 passed (classifier, size cap, and the worker connector copy across multiple chunks).
• The customer's original file (not committed) was checked in python:3.12-slim-trixie (libmagic 5.46, the backend's base image). Before: octet-stream → rejected. After: classify_file_path and classify_stream both return xlsx, and the bytes come through unchanged.
• Dev env (deepak-unstract-dev), before this change: the original ERP file sent to an API deployment → Rejecting file 'orig.xlsx' with unsupported MIME type 'application/octet-stream'; control workbook → Success.
• Dev env, after this change (snapshot.1793689200): the same original ERP file → Success (passed the gate and was extracted by LLMWhisperer); control workbook → Success. The ERP file was re-sent once a minute, and the result switched from rejection to Success when the new backend pod started serving. Prompt Studio upload in the UI not yet checked.

Screenshots

Checklist

I have read and understood the Contribution Guidelines.