A Dynamic Memory Managing List class?

[Ryex]

I came with an idea on how to keep memory usage down

the data objects we load are serialized as an array or a hash and when we load say the actors data we load a massive list of objects.

now we will rarely need access to more than 10 or 20 of these actor objects at once but the list could go into the 1000's in terms of size.
so I came up with this idea.

A list class that is "aware" of its contents but only keeps a necessary amount of it's objects in memory at once. when we load the initial list of data we would pass that list to this class
it would then keep at most say 100 objects in the list actually loaded in memory. the rest it would dump to tempfiles to be loaded on request. it would also have a way of detecting if there was a reference outside of itself to an object it contained. if there was no reference outside of the list that object would be a target for dumping to a temp file. if an object was requested form the list but wasn't actually loaded it would load it form the temp files. the class would probably keep the files it was storing the objects in open until the dynamic memory list was disposed.

such a class would have to be signed carefully to keep performance in mind but if done correctly we could manage our memory usage much better. assuming that it actually becomes necessary to manage the memory

[ForeverZer0]

That does sound like a something to look into if we need to. I would say get a more complete build going that has everything loaded into and see how the memory usage is before implementing such a system, like you said.

We can also just simply not load everything to begin with. Simply initialize arrays with None elements as placeholders for the size, and load them only when necessary.

[Blizzard]

I agree with F0. We should do optimizations after we are done.

I know exactly how this can be implemented. We make a new class that is derived from "dict". Then we simply define __set_item__ and __get_item__ in such a way that it doesn't directly access the data but only loads the data requested with a configurable cache size (we can make it 100). Do you know how page files work? It's basically the same concept. I can implement this class within a few minutes and you just have to replace the normal list you use for the objects.