The Problem

In a recent engagement with a customer, due to the nature of the engagement, there was a need to deal with config values coming from multiple sources. Actual secrets were stored in Azure KeyVault, some other config values were stored in JSON files or environment variables. In this kind of scenario, it is tempting to entangle the utility classes for each source of config data with business logic but this could quickly lead to spaghetti code.

In this article, we dive into a strategy for abstracting the configuration logic away from business logic. This article uses Python as an example but the pattern can be extended to other languages.

The Solution

Using the provider pattern, we create a provider class for each source of config data, ensuring that they all implement a common interface. For instance, a project which needs to read config data from a KeyVault, a JSON file and environment variables would have three provider classes – a KeyVaultProvider, a JSONProvider class and an AppEnvProvider class. Each provider class encapsulates the logic for accessing data from its source. We also include an InMemoryProvider class which allows us to set config values in memory.

A Configuration utility class also needs to be created. This class instantiates and stores the providers and is also responsible for calling each provider’s get_value method as appropriate. The Configuration class itself exposes two methods – get_value and set_value which can be used in the application code to fetch and set config values as necessary.

The Implementation

First, we create the AbstractConfigurationProvider using the inbuilt Abstract Base Classes from Python

class AbstractConfigurationProvider(ABC):
    """Abstract Provider class for configurations."""

    @abstractmethod
    def get_value(self, key: str):
        """Abstract method for all providers.

        Args
        ----
            key (str): string representing key to return value for
        """
        pass

Next, we create the concrete classes for each config source.

For KeyVault,

class KeyVaultConfigurationProvider(AbstractConfigurationProvider):

    kv_uri: str
    config: dict = {}

    def __init__(self, name: str) -> None:
        """Initialize the class."""
        self.kv_uri = f"https://{name}.vault.azure.net"

    def get_value(self, key: str):
        """Retrieve the value of the secret given the key."""
        try:
            if key in self.config:
                return self.config[key]
            credential = DefaultAzureCredential()
            client = SecretClient(vault_url=self.kv_uri, credential=credential)
            value = client.get_secret(key).value
            self.config[key] = value
            return value
        except ResourceNotFoundError:
            return None

For JSON Provider,

class JsonConfigurationProvider(AbstractConfigurationProvider):
    """Concrete Provider class for JSON files, can be extended by derivation."""

    def __init__(self, file_path: str):
        self.logger = logging.getLogger(__name__)
        self.file_contents = self._load_configuration(file_path)

    def get_value(self, key: str):
        """Get value."""
        if key in self.file_contents:
            return self.file_contents[key]
        return None

    def _load_configuration(self, file_path: str) -> dict:
        """Load JSON Configuration."""

        if not Path(file_path).exists():
            self.logger.error("JSON configuration file not found: %s", file_path)
            raise FileNotFoundError(f"JSON configuration file not found: {file_path}")

        try:
            with Path(file_path).open("r") as config_file:
                return json.load(config_file)
        except JSONDecodeError:
            self.logger.exception(
                "JSON decoding error for configuration file %s", file_path
            )
            raise

For the InMemory Provider,

class InMemoryConfigurationProvider(AbstractConfigurationProvider):
    """Concrete Provider class for In-memory config files."""

    _config: dict

    def __init__(self) -> None:
        self._config = {}

    def get_value(self, key: str):
        """Return the value of the key from the config."""
        try:
            return self._config[key]
        except KeyError:
            return None

    def set_value(self, key: str, value: str):
        """Set the value of the config."""
        self._config[key] = value

For the environment variables provider,

class AppEnvConfigurationProvider(AbstractConfigurationProvider):
    """Concrete Provider class for os.env."""

    def get_value(self, key: str):
        return os.getenv(key)

Finally, we create the actual Configuration class,

class Configuration:
    """Utility class to return all config values."""

    _providers: Any = None
    _in_memory_provider = InMemoryConfigurationProvider()

    @staticmethod
    def initialize():
        """Initialize the providers."""

        if Configuration._providers is None:
            json_config_file_path = # path to json config file
            Configuration._providers = []
            Configuration._providers.append(Configuration._in_memory_provider)
            Configuration._providers.append(AppEnvConfigurationProvider())
            Configuration._providers.append(JsonConfigurationProvider(json_config_file_path))
            kv_name = Configuration.get_value(key="KEYVAULT_NAME", nullable=True)
            if kv_name is not None:
                Configuration._providers.append(KeyVaultConfigurationProvider(kv_name))

    @staticmethod
    def set_value(key: str, value: str):
        """Set a config value in memory."""
        Configuration._in_memory_provider.set_value(key, value)

    @staticmethod
    def get_value(key: str, nullable=False):
        """Return the value of the key from any of the providers."""
        Configuration.initialize()
        value = None
        """
            This implementation returns the first available value. 
            In other scenarios where there is a different requirement (e.g same key existing in multiple providers),
            the implementation can be modified to meet the requirement.
        """
        for provider in Configuration._providers:
            value = provider.get_value(key)
            if value is not None:
                return value
        if nullable:
            return None

        raise ConfigurationError(f"Config value does not exist for {key}")

With this, config values can be accessed anywhere else in the code by simply calling

config_value = Configuration.get("KEY_TO_CONFIG")

Final Thoughts

This approach to fetching config values especially in non-traditional software allows for easy extensibility and ensures adherence to DRY principles. Config data from a different source (e.g an XML file) can be added without much modification to the existing codebase.

This pattern was useful to us and we hope you find it useful as well.

Note: the picture that illustrates this article was generated by Open AI.