Re: FM 2.8 Proposal : Fracturing, Cache, Rigidbody

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: FM 2.8 Proposal : Fracturing, Cache, Rigidbody

Sybren A. Stüvel
This is a response to
https://lists.blender.org/pipermail/bf-committers/2018-February/049157.html
(I removed it from my mail server, so I can't hit 'reply' directly)


On 26-02-2018 22:54, Brecht van Lommel wrote:

> Thanks for the explanation. Overall I agree with the approach.
>
> It would be very helpful if this kind of functionality was submitted for
> review and integration in smaller parts. For example:
> [...]
> * Add Alembic support to point caches for physics systems. (could even
> replace custom point cache format)

Alembic is stricly a write-once file format [1]. This means that we
cannot update a part of a file; say the file contains frame range 1-250,
we won't be able to re-simulate frames 150-250 and update the file for
that. It can only be "updated" by reading the old file, updating its
contents where necessary, and writing to another file. Furthermore, an
Alembic file cannot be read until it is properly written *and closed*.

With this info about Alembic, what do you think about using Alembic
files for such caches? I'm not yet all that familiar with the current
physics caching system, but I do know it creates a file for every frame,
making it trivial to update frames. Is this actually used, though? Or
are we always sequentially writing to the cache?

We could work around these limitations of Alembic by writing continuous
frame sequences to a single file (so if you simulate 1-100 and then
press ESC, it would write to sim-001-100.abc), which can then be read
when replaying those frames. Simulating other ranges would then save to
other Alembic files (e.g. sim-101-140.abc). This could potentially
produce many small files, though, or cause problems where objects exists
in one file but not the other. Then again, we could have code to merge
sequences of Alembic files.

If not Alembic, what would be a good cache file format? Ideally I'd like
to use something that's already proven technology, rather than thinking
up something ourselves.

[1]
https://groups.google.com/d/msg/alembic-discussion/X-2ue86pw5g/Z2fZpW1eAgAJ


--
Sybren A. Stüvel

https://stuvelfoto.nl/
https://stuvel.eu/

_______________________________________________
Bf-committers mailing list
[hidden email]
https://lists.blender.org/mailman/listinfo/bf-committers
Reply | Threaded
Open this post in threaded view
|

Re: FM 2.8 Proposal : Fracturing, Cache, Rigidbody

Brecht Van Lommel-4
For baking we normally simulate all frames, and even when we don't the cost
of merging two Alembic files is probably acceptable. However for caching
while doing animation playback this is not the case. We can simulate some
frames, play back the frames simulated so far, and then continue simulating
more frames. So that makes a single Alembic file not a great fit.

I think the main motivation to use Alembic is that the data structures are
more complicated than we had before, and our custom point cache format is
quite basic. Previously we just had to store vertex position/velocity/..
for every frame. With fracture we need to store the fractured mesh and its
topology, as well as the transforms per island. As I understand such
islands can be dynamically fractured off, so the mesh topology and active
islands can change over time. Combine that with the requirement of fast
random access for a given frame, things get quite complicated.

So I'm not sure what the right solution is to be honest. For fast random
read/write something like this could work:
* For every frame, write a point cache file in the same format we already
use. This would contain an array of island index, island transform, and any
other time varying data about active islands.
* In a separate file, store list of islands, their topology and any other
static data. When reading a frame, we'd read just the active islands. When
new islands are created during the sim, we could append to this file.

It would be better if we could a standard file format though, but which
ones support appending? Alternatively we could just say, fracture modifiers
is not something that works realtime anyway, so if we have to accept the
overhead of rewriting the same Alembic file many times it might not be that
bad.


On Tue, Jun 12, 2018 at 6:00 PM dr. Sybren A. Stüvel <[hidden email]>
wrote:

> This is a response to
> https://lists.blender.org/pipermail/bf-committers/2018-February/049157.html
> (I removed it from my mail server, so I can't hit 'reply' directly)
>
>
> On 26-02-2018 22:54, Brecht van Lommel wrote:
>
> > Thanks for the explanation. Overall I agree with the approach.
> >
> > It would be very helpful if this kind of functionality was submitted for
> > review and integration in smaller parts. For example:
> > [...]
> > * Add Alembic support to point caches for physics systems. (could even
> > replace custom point cache format)
>
> Alembic is stricly a write-once file format [1]. This means that we
> cannot update a part of a file; say the file contains frame range 1-250,
> we won't be able to re-simulate frames 150-250 and update the file for
> that. It can only be "updated" by reading the old file, updating its
> contents where necessary, and writing to another file. Furthermore, an
> Alembic file cannot be read until it is properly written *and closed*.
>
> With this info about Alembic, what do you think about using Alembic
> files for such caches? I'm not yet all that familiar with the current
> physics caching system, but I do know it creates a file for every frame,
> making it trivial to update frames. Is this actually used, though? Or
> are we always sequentially writing to the cache?
>
> We could work around these limitations of Alembic by writing continuous
> frame sequences to a single file (so if you simulate 1-100 and then
> press ESC, it would write to sim-001-100.abc), which can then be read
> when replaying those frames. Simulating other ranges would then save to
> other Alembic files (e.g. sim-101-140.abc). This could potentially
> produce many small files, though, or cause problems where objects exists
> in one file but not the other. Then again, we could have code to merge
> sequences of Alembic files.
>
> If not Alembic, what would be a good cache file format? Ideally I'd like
> to use something that's already proven technology, rather than thinking
> up something ourselves.
>
> [1]
> https://groups.google.com/d/msg/alembic-discussion/X-2ue86pw5g/Z2fZpW1eAgAJ
>
>
> --
> Sybren A. Stüvel
>
> https://stuvelfoto.nl/
> https://stuvel.eu/
>
> _______________________________________________
> Bf-committers mailing list
> [hidden email]
> https://lists.blender.org/mailman/listinfo/bf-committers
>
_______________________________________________
Bf-committers mailing list
[hidden email]
https://lists.blender.org/mailman/listinfo/bf-committers
Reply | Threaded
Open this post in threaded view
|

Re: FM 2.8 Proposal : Fracturing, Cache, Rigidbody

Jacob Merrill
what about caching a volume ?

use round / position to get the volume adress that ownes the thing

this way we have a fixed size array of lists of chunks each space ownes?

On Tue, Jun 12, 2018, 10:29 AM Brecht Van Lommel <[hidden email]>
wrote:

> For baking we normally simulate all frames, and even when we don't the cost
> of merging two Alembic files is probably acceptable. However for caching
> while doing animation playback this is not the case. We can simulate some
> frames, play back the frames simulated so far, and then continue simulating
> more frames. So that makes a single Alembic file not a great fit.
>
> I think the main motivation to use Alembic is that the data structures are
> more complicated than we had before, and our custom point cache format is
> quite basic. Previously we just had to store vertex position/velocity/..
> for every frame. With fracture we need to store the fractured mesh and its
> topology, as well as the transforms per island. As I understand such
> islands can be dynamically fractured off, so the mesh topology and active
> islands can change over time. Combine that with the requirement of fast
> random access for a given frame, things get quite complicated.
>
> So I'm not sure what the right solution is to be honest. For fast random
> read/write something like this could work:
> * For every frame, write a point cache file in the same format we already
> use. This would contain an array of island index, island transform, and any
> other time varying data about active islands.
> * In a separate file, store list of islands, their topology and any other
> static data. When reading a frame, we'd read just the active islands. When
> new islands are created during the sim, we could append to this file.
>
> It would be better if we could a standard file format though, but which
> ones support appending? Alternatively we could just say, fracture modifiers
> is not something that works realtime anyway, so if we have to accept the
> overhead of rewriting the same Alembic file many times it might not be that
> bad.
>
>
> On Tue, Jun 12, 2018 at 6:00 PM dr. Sybren A. Stüvel <[hidden email]>
> wrote:
>
> > This is a response to
> >
> https://lists.blender.org/pipermail/bf-committers/2018-February/049157.html
> > (I removed it from my mail server, so I can't hit 'reply' directly)
> >
> >
> > On 26-02-2018 22:54, Brecht van Lommel wrote:
> >
> > > Thanks for the explanation. Overall I agree with the approach.
> > >
> > > It would be very helpful if this kind of functionality was submitted
> for
> > > review and integration in smaller parts. For example:
> > > [...]
> > > * Add Alembic support to point caches for physics systems. (could even
> > > replace custom point cache format)
> >
> > Alembic is stricly a write-once file format [1]. This means that we
> > cannot update a part of a file; say the file contains frame range 1-250,
> > we won't be able to re-simulate frames 150-250 and update the file for
> > that. It can only be "updated" by reading the old file, updating its
> > contents where necessary, and writing to another file. Furthermore, an
> > Alembic file cannot be read until it is properly written *and closed*.
> >
> > With this info about Alembic, what do you think about using Alembic
> > files for such caches? I'm not yet all that familiar with the current
> > physics caching system, but I do know it creates a file for every frame,
> > making it trivial to update frames. Is this actually used, though? Or
> > are we always sequentially writing to the cache?
> >
> > We could work around these limitations of Alembic by writing continuous
> > frame sequences to a single file (so if you simulate 1-100 and then
> > press ESC, it would write to sim-001-100.abc), which can then be read
> > when replaying those frames. Simulating other ranges would then save to
> > other Alembic files (e.g. sim-101-140.abc). This could potentially
> > produce many small files, though, or cause problems where objects exists
> > in one file but not the other. Then again, we could have code to merge
> > sequences of Alembic files.
> >
> > If not Alembic, what would be a good cache file format? Ideally I'd like
> > to use something that's already proven technology, rather than thinking
> > up something ourselves.
> >
> > [1]
> >
> https://groups.google.com/d/msg/alembic-discussion/X-2ue86pw5g/Z2fZpW1eAgAJ
> >
> >
> > --
> > Sybren A. Stüvel
> >
> > https://stuvelfoto.nl/
> > https://stuvel.eu/
> >
> > _______________________________________________
> > Bf-committers mailing list
> > [hidden email]
> > https://lists.blender.org/mailman/listinfo/bf-committers
> >
> _______________________________________________
> Bf-committers mailing list
> [hidden email]
> https://lists.blender.org/mailman/listinfo/bf-committers
>
_______________________________________________
Bf-committers mailing list
[hidden email]
https://lists.blender.org/mailman/listinfo/bf-committers