Conversation
…oading Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
|
@copilot rewrite line based and raw token pointer scanning/parsing to byte-stream based. see https://raw.githubusercontent.com/lighttransport/tinyusdz/refs/heads/release/src/stream-reader.hh for the reference. rewrite IS_NEW_LINE, IS_SPACE, parse***, strspn, etc. to use the byte-stream based token parsing(add eof guard for each token consumption) |
Replace safeGetline + raw const char* token pointer scanning with a StreamReader class that wraps (buf, len, idx) with bounds-checked peek/advance/match operations. Every byte access is guarded by an EOF check. Key changes: - Add StreamReader class for bounds-checked text parsing - Add sr_* parse functions that use StreamReader instead of const char** - Add LoadMtlInternal, LoadObjInternal, LoadObjWithCallbackInternal using StreamReader; public APIs wrap these - mmap paths construct StreamReader directly from mapped memory - Remove safeGetline (no longer needed) - Keep old parse functions for backward compat with public ParseTextureNameAndOption API - Keep IS_SPACE, IS_NEW_LINE, IS_DIGIT macros for old parse functions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ked parsing - Add StreamReader class wrapping (buf, length) with EOF-guarded peek/advance/match - Add sr_* parse functions using StreamReader instead of const char** pointers - Add LoadMtlInternal, LoadObjInternal, LoadObjWithCallbackInternal taking StreamReader - mmap paths construct StreamReader directly from mapped memory - Remove safeGetline (no longer needed) - Keep old parse functions + macros for ParseTextureNameAndOption backward compat Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Implemented in f2c657b and c57ed2c. The parser is now byte-stream based using a Key changes:
|
There was a problem hiding this comment.
Pull request overview
This PR rewrites the OBJ/MTL parser from line-based (safeGetline + raw const char* token scanning) to a byte-stream-based approach using a new StreamReader class, and adds opt-in memory-mapped I/O support (TINYOBJLOADER_USE_MMAP).
Changes:
- Introduces
StreamReaderclass with bounds-checked byte access, and rewrites all parse functions assr_*variants consumingStreamReaderinstead ofconst char**pointers. - Converts
LoadObj,LoadMtl, andLoadObjWithCallbackto thin wrappers over newLoadObjInternal,LoadMtlInternal, andLoadObjWithCallbackInternalhelpers. - Adds mmap support for both POSIX and Windows for both
.objand.mtlfiles, with tests for the new paths.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
tiny_obj_loader.h |
Core rewrite: StreamReader class, all sr_* parse functions, mmap infrastructure for Windows/POSIX, internal helper functions, and membuf/imemstream dead code |
tests/tester.cc |
Two new tests: test_mmap_and_standard_load_agree (compares mmap vs stream results) and test_load_from_memory_buffer (verifies in-memory parsing consistency) |
Comments suppressed due to low confidence (1)
tiny_obj_loader.h:3165
- The mmap open/fstat/map/parse/unmap pattern is duplicated across six call sites: twice in
MaterialFileReader::operator()(multi-path loop and single-path branch, each for both Windows and POSIX) and twice inLoadObj(Windows and POSIX). Extracting the common logic into a static helper (e.g.,LoadMtlFromFileorLoadObjFromFile) would significantly reduce the code surface and the risk of the two copies diverging.
#ifdef TINYOBJLOADER_USE_MMAP
#if defined(_WIN32)
{
HANDLE hFile =
CreateFileA(filepath.c_str(), GENERIC_READ, FILE_SHARE_READ, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile == INVALID_HANDLE_VALUE) continue;
LARGE_INTEGER fileSize;
if (!GetFileSizeEx(hFile, &fileSize)) {
CloseHandle(hFile);
continue;
}
const size_t file_size = static_cast<size_t>(fileSize.QuadPart);
if (file_size == 0) {
CloseHandle(hFile);
StreamReader empty_sr("", 0);
LoadMtlInternal(matMap, materials, empty_sr, warn, err);
return true;
}
HANDLE hMapping =
CreateFileMappingA(hFile, NULL, PAGE_READONLY, 0, 0, NULL);
if (hMapping == NULL) {
CloseHandle(hFile);
continue;
}
const char *mmap_data = static_cast<const char *>(
MapViewOfFile(hMapping, FILE_MAP_READ, 0, 0, 0));
if (!mmap_data) {
CloseHandle(hMapping);
CloseHandle(hFile);
continue;
}
{
StreamReader sr(mmap_data, file_size);
LoadMtlInternal(matMap, materials, sr, warn, err);
}
UnmapViewOfFile(mmap_data);
CloseHandle(hMapping);
CloseHandle(hFile);
return true;
}
#else // POSIX mmap
{
int fd = open(filepath.c_str(), O_RDONLY);
if (fd == -1) continue;
struct stat sb;
if (fstat(fd, &sb) != 0) {
close(fd);
continue;
}
const size_t file_size = static_cast<size_t>(sb.st_size);
if (file_size == 0) {
close(fd);
StreamReader empty_sr("", 0);
LoadMtlInternal(matMap, materials, empty_sr, warn, err);
return true;
}
void *mapped = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);
if (mapped == MAP_FAILED) continue;
const char *mmap_data = static_cast<const char *>(mapped);
{
StreamReader sr(mmap_data, file_size);
LoadMtlInternal(matMap, materials, sr, warn, err);
}
munmap(mapped, file_size);
return true;
}
#endif // _WIN32
#else // !TINYOBJLOADER_USE_MMAP
std::ifstream matIStream(filepath.c_str());
if (matIStream) {
LoadMtl(matMap, materials, &matIStream, warn, err);
return true;
}
#endif // TINYOBJLOADER_USE_MMAP
}
std::stringstream ss;
ss << "Material file [ " << matId
<< " ] not found in a path : " << m_mtlBaseDir << "\n";
if (warn) {
(*warn) += ss.str();
}
return false;
} else {
std::string filepath = matId;
#ifdef TINYOBJLOADER_USE_MMAP
#if defined(_WIN32)
{
HANDLE hFile =
CreateFileA(filepath.c_str(), GENERIC_READ, FILE_SHARE_READ, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile != INVALID_HANDLE_VALUE) {
LARGE_INTEGER fileSize;
if (GetFileSizeEx(hFile, &fileSize)) {
const size_t file_size = static_cast<size_t>(fileSize.QuadPart);
if (file_size == 0) {
CloseHandle(hFile);
StreamReader empty_sr("", 0);
LoadMtlInternal(matMap, materials, empty_sr, warn, err);
return true;
}
HANDLE hMapping =
CreateFileMappingA(hFile, NULL, PAGE_READONLY, 0, 0, NULL);
if (hMapping != NULL) {
const char *mmap_data = static_cast<const char *>(
MapViewOfFile(hMapping, FILE_MAP_READ, 0, 0, 0));
if (mmap_data) {
{
StreamReader sr(mmap_data, file_size);
LoadMtlInternal(matMap, materials, sr, warn, err);
}
UnmapViewOfFile(mmap_data);
CloseHandle(hMapping);
CloseHandle(hFile);
return true;
}
CloseHandle(hMapping);
}
}
CloseHandle(hFile);
}
}
#else // POSIX mmap
{
int fd = open(filepath.c_str(), O_RDONLY);
if (fd != -1) {
struct stat sb;
if (fstat(fd, &sb) == 0) {
const size_t file_size = static_cast<size_t>(sb.st_size);
if (file_size == 0) {
close(fd);
StreamReader empty_sr("", 0);
LoadMtlInternal(matMap, materials, empty_sr, warn, err);
return true;
}
void *mapped = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);
if (mapped != MAP_FAILED) {
const char *mmap_data = static_cast<const char *>(mapped);
{
StreamReader sr(mmap_data, file_size);
LoadMtlInternal(matMap, materials, sr, warn, err);
}
munmap(mapped, file_size);
return true;
}
} else {
close(fd);
}
}
}
#endif // _WIN32
#else // !TINYOBJLOADER_USE_MMAP
std::ifstream matIStream(filepath.c_str());
if (matIStream) {
LoadMtl(matMap, materials, &matIStream, warn, err);
return true;
}
#endif // TINYOBJLOADER_USE_MMAP
std::stringstream ss;
ss << "Material file [ " << filepath
<< " ] not found in a path : " << m_mtlBaseDir << "\n";
if (warn) {
(*warn) += ss.str();
}
return false;
}
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@copilot resolve conflicts nicely. basically accept both changes. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ent, BOM in callback, test fixes - Add UTF8ToWchar/LongPathW helpers for Windows long path and Unicode support - Add Windows long path support (#ifdef _WIN32) in non-mmap ifstream fallbacks - Add vt_w vector and optional w component parsing in LoadObjInternal - Fix texcoord_ws.swap(vt) -> texcoord_ws.swap(vt_w) corruption - Add BOM stripping to LoadObjWithCallbackInternal - Fix invalid-relative-texture-vertex-index.obj -> invalid-relative-texture-index.obj - Add test models: texcoord-w.obj, texcoord-w-mixed.obj, utf8-path-test.obj/.mtl - Add tests: texcoord_w, texcoord_w_mixed, utf8_path, long_path, callback_BOM Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
All changes from both release commits (#414, #415) have been manually integrated into our StreamReader-based branch in 5ef0c48:
All 51 tests pass. However, since I can't do |
…lease Keep StreamReader-based parsing from the harden branch for all conflicting regions: map_d parsing, mmap material loading, LoadObjInternal, texcoord w-component, BOM handling, and main parse loop. Retain mmap/memory-buffer tests in tester.cc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add column tracking (col_num_) to StreamReader, updated in all position-advancing methods (get, advance, skip_space, etc.) - Add current_line_text(), format_error(filename, msg) for clang-style diagnostics (file:line:col with source line and caret) - Add error stack to StreamReader (push_error, has_errors, get_errors) - Add error-reporting sr_parse overloads that return bool for success/failure: sr_parseInt, sr_parseReal, sr_parseReal2/3, sr_parseVertexWithColor (returns -1 on error) - Wire error-reporting overloads into LoadMtlInternal (Ka, Kd, Ks, Ni, Ns, illum, d, Tr, PBR properties) with early return on parse failure; change LoadMtlInternal from void to bool - Wire error-reporting overloads into LoadObjInternal (v, vn, vt, vw) with early return on parse failure - Upgrade existing error messages (face, line, points, skin weight) to use sr.format_error() for consistent clang-style output - Pass filename through call chain: LoadObj -> LoadObjInternal, MaterialFileReader -> LoadMtlInternal - Add warning_context.filename for fixIndex warnings - Add 6 new tests: column tracking, format_error output, error stack, malformed vertex/mtl errors, backward compatibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move warning_context initialization (warn, filename) before the main parse loop in LoadObjInternal; only update line_number per iteration. Avoids a std::string copy of filename on every face/line/points line. - Replace test_parse_error_backward_compat (was just TEST_CHECK(true)) with an actual round-trip test: parse valid OBJ, verify no errors and correct vertex count. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@copilot code review[agent] Review changes with GPT-5.3-Codex |
…m naming, extract sr_skipTagField helper, refactor mmap to RAII MappedFile Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
…erals Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Addressed in Changes made for each low severity item:
All 51 tests pass (both standard and |
|
@copilot run comprehensive review again, now GPT-5.3-Codex |
Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Ran a comprehensive GPT-5.3-Codex review on Findings:
Also checked recent Actions runs/logs for this branch; latest Unit Tests run is passing, with no failed jobs in recent action_required runs. No UI changes in this PR (no screenshot). |
@copilot do all these |
Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Implemented in Done all three items:
Validation:
No UI changes in this PR (no screenshot). |
|
@copilot deeply review. make this time final round. |
Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Final deep review completed and fixes applied in I found and fixed a high-severity callback/material-loading issue:
Validation:
No UI changes in this PR (no screenshot). |
|
@copilot deeper review again. final stage 2. maximum effort. ultrathink. |
… parity Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Completed stage-2 deep review and applied fixes in Key findings and fixes:
Validation:
No UI changes in this PR (no screenshot). |
|
@copilot further deep code review. final stage 3 |
…ze check Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Completed final stage-3 deep review and pushed fixes in Applied improvements:
Validation:
No UI changes in this PR (no screenshot). |
4009096646)make check💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.