I am designing a multi tenant workload automation software (software to run Jobs automatically). For this I am creating a default Job configuration class. Configurations specified in this class will be applied to all types of Jobs by default.
Tenants (owners of Jobs) can opt to override these default configurations for their specific class of Jobs.
For example:
# Default configurations for all types of Jobs
class DefaultConfigurations:
def __init__(self, job_defintion):
self.job_state_database = DEFAULT_DB
self.job_definition_repository_type = DEFAULT_REPO
....
# there will be 100's of configurations like this.
Now if some tenant wants to override the default application configuration for their specific type of jobs, they can inherit from DefaultConfiguration class and override the configurations that they want to override.
For example:
# These overridden configurations will be applied to all the HiveJobs.
class HiveJobs(DefaultConfigurations):
def __init__(self, job_definition):
self.job_state_database = "sql"
self.job_definition_repository_type = "svn"
# These overridden configurations will be applied to all the SparkJobs.
class SparkJobs(DefaultConfigurations):
def __init__(self, job_definition):
self.job_state_database = "MongoDb"
if (job_definition.technology == "Old")
self.job_state_database = "sql"
For all other types of jobs, default configurations will be used.
Individual jobs too have their definitions (mentioned in XML form). In an individual job definition XML file, class of job is also specified. For example, Hive Job will specify its class as "hive" in its definition.
Example of job_definition file for one of the hive jobs:
<job_definition>
name hello_world_from_hive
class hive
command echo "hello world from Hive"
cron_schedule 5 4 * * *
</job_defintion>
At runtime, Job Executor will check the class of Job that is specified in its definition file and pick the configuration class accordingly (for example: DefaultConfigurations, HiveJobs or SparkJobs in the example above).
Job executor will construct a job_definition object from XML file and pass that Job definition object to the corresponding configuration class to get the final configurations that are needed to execute this job. This is needed so that some configurations can be added/removed based on some run time parameters too. Please note that the preference will be given to configurations overridden in individual Job definition file.
I am not sure if the above way is the best way to write such configuration files in Python.
from Application configuration files in Python
No comments:
Post a Comment