Skip to content

Push based writer#54

Open
gatesn wants to merge 1 commit intodevelopfrom
ngates/push-based-writer
Open

Push based writer#54
gatesn wants to merge 1 commit intodevelopfrom
ngates/push-based-writer

Conversation

@gatesn
Copy link
Copy Markdown
Contributor

@gatesn gatesn commented Apr 16, 2026

Change the Vortex writer from streaming pull-based to push-based by default to better align with ecosystem APIs, and to allow for introspection of incremental progress to support file rotation and other similar use-cases.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Comment on lines +204 to +205
/// chunk. It must not wait for all segment I/O derived from this chunk to reach the underlying
/// file sink.
Copy link
Copy Markdown
Collaborator

@AdamGS AdamGS Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While obviously there's no way to enforce that data is persisted here, it does seem useful to the writer-facing API to be sure that the data isn't in memory anymore in some predictable way.

Copy link
Copy Markdown
Collaborator

@AdamGS AdamGS Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the idea here that a user can push multiple chunks and then call flush to be sure that there's no more buffering?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants