Python3: How to change the design of a class hierarchy to improve access to objects there buried?

I asked this question already at stackoverflow together with a serialization related part and at codereview for the design part only. Since the design related part receives no answers or comments on stack exchange and was off-topic on codereview I’d like to ask about it here.

The problem: I have a complicated class hierarchy in which the classes are similar to each other but every class contains a bunch of more or less complicated stateful variables. To give you an impression, please have a look at this minimal working example:

#!/usr/bin/env python3 # -*- coding: utf-8 -*-  import numpy as np   class OptimizableVariable:     """     Complicated stateful variable. It is intended to return mainly     floating point values. Status determines whether an optimization     later shall change the value ("V") or not ("F"). The status "P"     is for a variable which depends on a number of other variables.     """     def __init__(self, name="", status="F", **kwargs):         self.name = name         self.status = status         self.parameters = kwargs         self.eval_dict = {"F": self.eval_fixed,                           "V": self.eval_variable,                           "P": self.eval_pickup}      def eval_fixed(self):         return self.parameters.get("value", 0.0)      def eval_variable(self):         return self.parameters.get("value", 0.0)      def eval_pickup(self):         fun = self.parameters.get("function", None)         optvar_arguments = self.parameters.get("arguments", tuple())          if fun is not None:             call_args = tuple([ov.evaluate() for ov in optvar_arguments])             return fun(*call_args)      def evaluate(self):         return self.eval_dict[self.status]()      def setvalue(self, value):         if self.status == "F" or self.status == "V":             self.parameters["value"] = value   class OptimizableVariableContainer:     """     Class which contains several OptVar objects and nested OptVarContainer     classes. Is responsible for OptVar management of its sub-OptVarContainers     with their respective OptVar objects.     """     def __init__(self, name="", **kwargs):         self.name = name         for (key, value_dict) in kwargs.items():             setattr(self, key, OptimizableVariable(name=key, **value_dict))      def getAllOptVars(self):          def addOptVarToDict(var,                             dictOfOptVars={},                             idlist=[],                             reducedkeystring=""):             """             Accumulates optimizable variables in var and its linked objects.             Ignores ring-links and double links.              @param var: object to evaluate (object)             @param dictOfOptVars: optimizable variables found so far (dict of dict of objects)             @param idlist: ids of objects already evaluated (list of int)             @param reducedkeystring: to generate the dict key (string)             """              if id(var) not in idlist:                 idlist.append(id(var))                  if isinstance(var, OptimizableVariableContainer):                     for (k, v) in var.__dict__.items():                         newredkeystring = reducedkeystring + var.name + "."                         dictOfOptVars, idlist = addOptVarToDict(v,                                                                 dictOfOptVars,                                                                 idlist,                                                                 newredkeystring)                  elif isinstance(var, OptimizableVariable):                     newreducedkeystring = reducedkeystring + var.name                     dictOfOptVars[newreducedkeystring] = var                 # here other elifs are also possible, checking for                 # OptimizableVariables in lists or dicts within the                 # class instance              return dictOfOptVars, idlist          (dict_opt_vars, idlist) = addOptVarToDict(self, reducedkeystring="")         return dict_opt_vars      def getStatusVVariables(self):         """         Get all OptimizableVariables with status "V"         """         optvars = self.getAllOptVars()         return [ov for ov in optvars.values() if ov.status == "V"]      def getVValues(self):         """         Function to get all values of status "V" variables         into one large np.array. Interface to scipy.optimize functions.         """         return np.array([a.evaluate() for a in self.getStatusVVariables()])      def setVValues(self, x):         """         Function to set all values of active variables to the values         in the large np.array x. Interface to scipy.optimize functions.         """         for i, var in enumerate(self.getStatusVVariables()):             var.setvalue(x[i])    class LocalCoordinates(OptimizableVariableContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(LocalCoordinates, self).__init__(name=name,              **{"x": {"status": "F", "value": 0.},                 "y": {"status": "F", "value": 0.},                 "z": {"status": "F", "value": 0.}})   class Surface(OptimizableVariableContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(Surface, self).__init__(name=name,              **{"curvature": {"status": "F", "value": 0.0}})         self.lc = LocalCoordinates(name=name + "_lc")   class System(OptimizableVariableContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(System, self).__init__(name=name,                 **{"some_property": {"status": "F", "value": 1.0},                    "some_other_property": {"status": "F", "value": 0.0}})         self.surf1 = Surface(name="surf1")         self.surf2 = Surface(name="surf2")   def surf1_lc_on_circle(mysys):     """     Scalar function to verify that x, y of lc of surf1 are on a unit circle     """     return (mysys.surf1.lc.x.evaluate()**2 +             mysys.surf1.lc.y.evaluate()**2 - 1)**2      # optvars = mysys.getStatusVVariables()     # numpy_vector = np.asarray([ov.evaluate() for ov in optvars])   def main():     # creating OptimizableVariableContainer with some nested     # OptimizableVariableContainers.     my_sys = System(name="system")      # It is intended behaviour to access the OptimizableVariable objects via     # scoping within the class hierarchy.     print(my_sys.surf1.lc.x.evaluate())     my_sys.surf1.lc.x.parameters["value"] = 2.0     print(my_sys.surf1.lc.y.parameters)     my_sys.surf1.lc.z = OptimizableVariable(name="z", status="P",                                             function=lambda x, y: x**2 + y**2,                                             arguments=(my_sys.surf1.lc.x,                                                        my_sys.surf1.lc.y))     my_sys.surf1.lc.x.status = "V"     my_sys.surf1.lc.y.status = "V"     print(my_sys.surf1.lc.z.evaluate())      # The following construction is quite ugly:     # a) due to the dict: order is not fixed     # b) due to recursion (and maybe lexical sorting) it is not fast     # c) goal: hand over these optvars into some numerical optimization     # via a numpy array => should be fast     optvarsdict = my_sys.getAllOptVars()     for (key, ov) in optvarsdict.items():         print("%s(%s): %f" % (key, ov.status, ov.evaluate()))      # Optimization of status "V" variables due to a scalar optimization     # criterion is an important point in the code. This is done manually in     # the following:      print(surf1_lc_on_circle(my_sys))  # check value of scalar function (>0)     print(my_sys.getVValues())  # get numpy array of V variables     my_sys.setVValues(np.array([0., 1.]))  # this pair is on a unit circle     # Notice that since the values come from a dict, it is not guaranteed     # that the array is [x, y] nor is guaranteed that the order is preserved     print(surf1_lc_on_circle(my_sys))  # check value of scalar function (==0)      # Notice that my_sys got changed (which is intended behaviour):     # Notice also that my_sys.surf1.surf1_lc.z changed since it is of     # status "P" (also intended behaviour)     optvarsdict = my_sys.getAllOptVars()     for (key, ov) in optvarsdict.items():         print("%s(%s): %f" % (key, ov.status, ov.evaluate()))   if __name__ == "__main__":     main() 

The question is now: How to access the OptimizableVariable objects the best. I already thought about to use some kind of pool object and use some proxy within the class hierarchy to have a link to the pool. This pool should not be a singleton, since it should be possible to manage more than one pool at a time. Up to now the access to the OptimizableVariable variables within my own code is done via the getAllOptVars function. This is quite ugly and only considered temporary. Is there a better alternative to this function?

To summarize and clarify my goals:

  1. The class hierarchy is ok and reflects the model context of our problem
  2. The OptimizableVariables belong to each object in the hierarchy and should also stay there also due to model context
  3. The access and post processing of the OptimizableVariables (i.e. collecting them from the hierarchy and handing them over to an optimizer as a numpy array) is not considered to be optimal. There I need some suggestions on doing better (i.e. getting rid of isinstance and id queries).
  4. A “nice to have” would be: Decouple serialization of the OptimizableVariables and version management of classes from the object hierarchy (i.e. it would be nice to have an interface to make snap shots of certain objects in the hierarchy)
  5. Another “nice to have” would be: How to incorporate shared OptimizableVariables and “singleton” OptimizableVariables in this class hierarchy best?

I am aware that no unique solutions exist for these design problems, but I need further input on that. Thank you!

PS: If the minimal working example is too long and does not show the goal of the code, I’m sorry and I could reduce it.

Better way of access the stateful variables objects in the given class hierarchy

I asked this question already at stackoverflow together with a serialization related part. Since the design related part receives no answers or comments I’d like to have a review on this here.

The problem: I have a complicated class hierarchy in which the classes are similar to each other but every class contains a bunch of more or less complicated stateful variables. To give you an impression, please have a look at this minimal working example:

class OptVar:     """     Complicated stateful variable     """     def __init__(self, name="", **kwargs):         self.name = name         self.parameters = kwargs   class OptVarContainer:     """     Class which contains several OptVar objects and nested OptVarContainer     classes. Is responsible for OptVar management of its sub-OptVarContainers     with their respective OptVar objects.     """     def __init__(self, name="", **kwargs):         self.name = name         for (key, value_dict) in kwargs.items():             setattr(self, key, OptVar(name=key, **value_dict))      def getAllOptVars(self):          def addOptVarToDict(var,                             dictOfOptVars={},                             idlist=[],                             reducedkeystring=""):             """             Accumulates optimizable variables in var and its linked objects.             Ignores ring-links and double links.              @param var: object to evaluate (object)             @param dictOfOptVars: optimizable variables found so far (dict of dict of objects)             @param idlist: ids of objects already evaluated (list of int)             @param reducedkeystring: to generate the dict key (string)             """              if id(var) not in idlist:                 idlist.append(id(var))                  if isinstance(var, OptVarContainer):                     for (k, v) in var.__dict__.items():                         newredkeystring = reducedkeystring + var.name + "."                         dictOfOptVars, idlist = addOptVarToDict(v,                                                                 dictOfOptVars,                                                                 idlist,                                                                 newredkeystring)                  elif isinstance(var, OptVar):                     newreducedkeystring = reducedkeystring + var.name                     dictOfOptVars[newreducedkeystring] = var              return dictOfOptVars, idlist          (dict_opt_vars, idlist) = addOptVarToDict(self, keystring="")         return dict_opt_vars    class C(OptVarContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(C, self).__init__(name=name,                 **{"my_c_a": {"c1": 1, "c2": 2},                    "my_c_b": {"c3": 3, "c4": 4}})   class B(OptVarContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(B, self).__init__(name=name, **{"b": {"1": 1, "2": 2}})         self.c_obj = C(name="cname")   class A(OptVarContainer):     """     Specific implementation of class OptVarContainer     """     def __init__(self, name=""):         super(A, self).__init__(name=name,                 **{"a1": {"1": 1, "2": 2},                    "a2": {"a": "a", "b": "b"}})         self.b_obj = B(name="bname")   def main():     # creating OptVarContainer with some nested OptVarContainers.     my_a_obj = A(name="aname")      # It is intended behaviour to access the OptVar objects via     # scoping within the class hierarchy.     print(my_a_obj.b_obj.b.parameters)     my_a_obj.b_obj.b.parameters["2"] = 3     print(my_a_obj.b_obj.b.parameters)     print(my_a_obj.b_obj.c_obj.my_c_a.parameters["c1"])     my_a_obj.b_obj.c_obj.my_c_a.parameters["c1"] = 6     print(my_a_obj.b_obj.c_obj.my_c_a.parameters)      # This construction is quite ugly:     # a) due to the dict: order is not fixed     # b) due to recursion (and maybe lexical sorting) it is not fast     # c) goal: hand over these optvars into some numerical optimization     # via a numpy array => should be fast     optvarsdict = my_a_obj.getAllOptVars()     print(optvarsdict)   if __name__ == "__main__":     main() 

The question is now: How to access the OptVar objects the best. I already thought about to use some kind of pool object and use some proxy within the class hierarchy to have a link to the pool. This pool should not be a singleton, since it should be possible to manage more than one pool at a time. Up to now the access to the OptVar variables within my own code is done via the getAllOptVars function. This is quite ugly and only considered temporary. Is there a better alternative to this function?

To summarize and clarify my goals:

  1. The class hierarchy is ok and reflects the model context of our problem
  2. The OptVars belong to each object in the hierarchy and should also stay there also due to model context
  3. The access and post processing of the OptVars (i.e. collecting them from the hierarchy and handing them over to an optimizer as a numpy array) is not considered to be optimal. There I need some suggestions on doing better (i.e. getting rid of isinstance and id queries).
  4. A nice to have would be: Decouple serialization of the OptVars and version management from the object hierarchy

I am aware that no unique solutions exist for this design problem, but I need further input on that.

The full context of this question is given here but after some discussions I separated the design part.