Load file partially to avoid buffer overload

[kldjonge]

Is it possible to only load the .mat file partially?
When only wanting to extract a few variables it seems that still the whole .mat file is loaded. This results in a buffer overload error when the .mat file gets to big (e.g. when also the protected variables were stored for debugging purposes) which might be avoided when only the necessary data is loaded.

I got the inspiration from the optional 'variable_names' argument in de scipy.io.loadmat class.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html

[mwetter]

Did you check if DyMat supports this? Reading the .mat files is done through the DyMat library.

[kldjonge]

I checked indeed and it doesn't i'm afraid. DyMat uses the scipy.io.loadmat I mentioned but doesn't use the 'variables _names' option so I think it always loads the whole file.

[Walshest]

Hi @kldjonge ,
Did you find a solution to the buffer error? I have the same issue with larger files: #474

[kldjonge]

@Walshest, I haven't.
DyMat (and later BuildingsPy) should be updated so that the optional argument 'variables_names' of scipy.io.loadmat could be used.

[thorade]

https://stackoverflow.com/questions/70513151/opening-big-result-mat-file-raises-buffer-is-too-small-for-requested-array
jraedler/DyMat#4

[kldjonge]

@thorade, indeed, exporting a smaller .MAT file is what I ended up doing. Well, I first simulated and stored all variables (might need them for later analysis or used to check the internal workings of the model/controls) then I plotted the once I needed and exported a .MAT file with only those variables.

This wasn't convient though and is a lot of manual work for bath simulations. It would be nice to be able to load only a subset of the original .MAT file directly in Python. I haven't actually tried this, but as scipy.io.loadmat can do it, probably the fix is not that far away. Although I don't think there is any active developement for DyMat.

[Walshest]

Hi @thorade and @kldjonge ,
Thanks for all of the feedback and suggestions.
I can confirm that Variable selections as outlined here: https://stackoverflow.com/questions/70513151/opening-big-result-mat-file-raises-buffer-is-too-small-for-requested-array reduced the size of the simulation file a lot.

However there is still a limitation when we run much longer & complex Modelica simulation e.g. for months/ year of seasonal operation.

• It would be defiantly beneficial if the BuildingsPy was updated to support these larger files. e.g. only loading variables of interest or resampling of the variables.

[bravache]

Thanks @Walshest , I agree that it would probably be better to allow BuildingsPy to handle larger results. I will take a look at it.