![\"maintenance](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/02\/maintenance-window-database-migrations-0.jpg\")
<\/p>\nIn a recent project, we were tasked with implementing a database migration strategy.\nThe goal was to minimize the impact on our users and to ensure that during\nthe database migration database would not be corrupted with writes and reads\nin the process. This blog post will describe the strategy we implemented and\nhow we applied it.<\/p>\n
Database migrations can be a tricky task for any organization, especially when\nit comes to maintaining the availability of the system during the migration process.\nOne way to mitigate this risk is to implement a maintenance window strategy.<\/p>\n
In a maintenance window, the application is taken offline for maintenance or updates.\nThis approach enables IT teams to perform necessary updates or migrations\nwithout compromising the system’s availability for end users or risking a\ncorrupt database.<\/p>\n
As can be seen below in the diagram, the maintenance window strategy is\nimplemented by taking the system offline during the maintenance window and\nperforming the database migration. Once the migration is complete, the system\nis brought back online. During the maintenance window, the system is not\navailable for end users and any requests made to the system receive a 503\nhttp status code.<\/p>\n
<\/p>\n
The criteria for implementing a maintenance window strategy for database\nmigrations are:<\/p>\n
All source code in this article can be found here<\/a>.<\/p><\/blockquote>\n
For a real-world example, we will look at a maintenance window strategy implementation for\na FLASK application<\/a> with a SQL based database. Given that the application uses SQLAlchemy<\/a> and alembic<\/a> for\ndatabase migrations we will have the following components:<\/p>\n
Service context database model<\/h3>\n
The service context database model is a database model that is used to store the\ncurrent state of the service. This model is used to determine if the service is\nin maintenance mode or not. The model is defined as follows:<\/p>\n
Note: The service context database model can also be extended to store other\ninformation about the service such as the current version of the service.<\/p><\/blockquote>\n
from src.infrastructure.databases import sqlalchemy_db as db\r\nfrom src.infrastructure.models.model_extension import ModelExtension\r\n\r\nclass ServiceContext(db.Model, ModelExtension):\r\n __tablename__ = 'service_context'\r\n id = db.Column(db.Integer, primary_key=True)\r\n maintenance = db.Column(db.Boolean, default=False)<\/code><\/pre>\nWe use a database model to store the current state of the service because it allows\nus to store the state of the service persistently. This means that if the\nservice is restarted, the state of the service will be restored. Also, if the service\nis running on multiple instances, the state of the service will be consistent across\nall instances.<\/p>\n
Service context service<\/h3>\n
The service context service is a service that is used to interact with the service\ncontext database model. The service context service is defined as follows:<\/p>\n
from src.infrastructure.models import ServiceContext\r\nfrom src.infrastructure.databases import sqlalchemy_db as db\r\n\r\nclass ServiceContextService:\r\n\r\n def update(self, data):\r\n service_context = ServiceContext.query.first()\r\n\r\n if service_context is None:\r\n service_context = ServiceContext()\r\n\r\n service_context.update(db, data)\r\n return service_context\r\n\r\n def get_service_context(self):\r\n status = ServiceContext.query.first()\r\n\r\n if status is None:\r\n status = ServiceContext()\r\n status.save(db)\r\n\r\n return status<\/code><\/pre>\nMaintenance mode activation and deactivation<\/h3>\n
To activate of deactivate the maintenance mode, we can use a route as shown below:<\/p>\n
@blueprint.route('\/service-context', methods=['PATCH'])\r\n@post_data_required\r\n@inject\r\ndef update_service_context(\r\n json_data,\r\n service_context_service=Provide[\r\n DependencyContainer.service_context_service\r\n ]\r\n):\r\n validated_data = ServiceContextSchema().load(json_data)\r\n service_context = service_context_service.update(validated_data)\r\n return create_response(service_context, ServiceContextSchema)<\/code><\/pre>\nWhen creating this public route, make sure that you have a way to authenticate the user that makes the request in order to determine that the user has the necessary permissions.<\/p>\n
You can also create a management command with Flask cli<\/a> to\nactivate the maintenance mode from within the application:<\/p>\n
@app.cli.command(\"activate_maintenance_mode\")\r\ndef activate_maintenance_mode():\r\n service_context_service = app.container.service_context_service()\r\n service_context_service.update({\"maintenance\": True})\r\n logger.info(\"Maintenance mode activated\")\r\n\r\n@app.cli.command(\"deactivate_maintenance_mode\")\r\ndef deactivate_maintenance_mode():\r\n service_context_service = app.container.service_context_service()\r\n service_context_service.update({\"maintenance\": False})\r\n logger.info(\"Maintenance mode deactivated\")<\/code><\/pre>\nThis allows you to activate or deactivate the maintenance mode from the command line if\nyou have access to the server where the application is running.<\/p>\n
Maintenance mode check<\/h3>\n
We are using the
before_request<\/code> decorator to run the maintenance mode check before\neach request. The maintenance mode check is defined as follows:<\/p>\n@app.before_request\r\ndef check_for_maintenance():\r\n service_context_service = app.container.service_context_service()\r\n status = service_context_service.get_service_context()\r\n\r\n if not (\"maintenance\" in request.path or \"status\" in request.path):\r\n if status.maintenance:\r\n return jsonify(\r\n {\"message\": \"Service is currently enduring maintenance\"}\r\n ), 503<\/code><\/pre>\nWith this implementation, the service will return a 503 http status code for all\nrequests that are not made to the maintenance mode activation, deactivation\nor status routes.<\/p>\n
Applying migrations during maintenance window<\/h2>\n
You have several ways to apply migrations during a maintenance window.<\/p>\n
One way is to do the migration manually by running the migration commands from\nwithin the application. This is the simplest way to apply migrations and gives\nyou the most control over the migration process.<\/p>\n
Manual migrations<\/h3>\n
An example of applying the migration manually is shown below in a kubernetes cluster:<\/p>\n
\n
- Access a pod that has access to a database and has the maintenance window strategy implemented\n
kubectl exec -it <pod_name> -- \/bin\/bash<\/code><\/pre>\n<\/li>\n- Activate the maintenance mode\n
python manage.py activate_maintenance_mode<\/code><\/pre>\n<\/li>\n- Apply the migration\n
python manage.py db upgrade<\/code><\/pre>\n<\/li>\n- Deactivate the maintenance mode\n
python manage.py deactivate_maintenance_mode<\/code><\/pre>\n<\/li>\n<\/ol>\nPipeline based migration<\/h3>\n
This is a more advanced way to apply migrations during a maintenance window.\nThe pipeline below is an azure devops pipeline and an example of an azure\napp service deployment where a maintenance window is used to apply the migration.<\/p>\n
The pipeline is defined as follows:<\/p>\n
trigger:\r\n paths:\r\n include:\r\n - <service_path_from_rooth>\/migrations\/*\r\n\r\nvariables:\r\n production_web_app_service_url: \"https:\/\/<production_service_url>\"\r\n staging_web_app_service_url: \"https:\/\/<staging_web_app_service_url>\"\r\n projectRoot: $CI_PROJECT_DIR\r\n vmImageName: ubuntu-latest\r\n pythonVersion: 3.8\r\n isMain: $[eq(variables['Build.SourceBranch'], 'refs\/heads\/main')]\r\n\r\nstages:\r\n- stage: \"deploy_stable\"\r\n condition: and(always(), eq(variables['isMain'], True))\r\n jobs:\r\n - job: \"build_web_app\"\r\n displayName: \"Build web app\"\r\n pool:\r\n vmImage: $(vmImageName)\r\n steps:\r\n - task: UsePythonVersion@0\r\n inputs:\r\n versionSpec: '$(pythonVersion)'\r\n - script: |\r\n python -m venv venv\r\n source venv\/bin\/activate\r\n python -m pip install --upgrade pip\r\n pip install setup\r\n pip install --target=\".\/.python_packages\/lib\/site-packages\" -r .\/requirements.txt\r\n workingDirectory: $(projectRoot)\r\n displayName: \"Install requirements\"\r\n - task: ArchiveFiles@2\r\n inputs:\r\n rootFolderOrFile: '$(Build.SourcesDirectory)'\r\n includeRootFolder: false\r\n archiveType: 'zip'\r\n archiveFile: '$(Build.ArtifactStagingDirectory)\/Application$(Build.BuildId).zip'\r\n replaceExistingArchive: true\r\n - publish: $(Build.ArtifactStagingDirectory)\/Application$(Build.BuildId).zip\r\n displayName: 'Upload package'\r\n artifact: drop\r\n - script: |\r\n curl -X PATCH -H \"Content-Type: application\/json\" -d '{\"maintenance\": true}' $(production_web_app_service_url)\/service-context\r\n name: activate_maintenance_mode_production\r\n displayName: Activate maintenance mode on production\r\n - task: AzureWebApp@1\r\n displayName: \"Deploy web app to staging\"\r\n inputs:\r\n azureSubscription: '<Azure service connection>'\r\n appType: webAppLinux\r\n appName: '<name of web app>'\r\n deployToSlotOrASE: true\r\n resourceGroupName: '<name of resource group>'\r\n slotName: staging\r\n - script: |\r\n curl -X PATCH -H \"Content-Type: application\/json\" -d '{\"maintenance\": true}' $(staging_web_app_service_url)\/service-context\r\n name: activate_maintenance_mode_staging\r\n displayName: Activate maintenance mode on staging\r\n - task: AzureKeyVault@2\r\n displayName: Load key vault db connection string\r\n inputs:\r\n connectedServiceName: azure-keyvault-connection\r\n keyVaultName: $(keyVaultName)\r\n secretsFilter: '*'\r\n - script: |\r\n > .env\r\n echo SQLALCHEMY_DATABASE_URI=\"$(SQLALCHEMY_DATABASE_URI)\" >> .env\r\n name: setup_env_file\r\n displayName: Setup env file\r\n - script: |\r\n python manage.py db upgrade\r\n name: apply_migrations\r\n displayName: Apply migrations\r\n - task: AzureAppServiceManage@0\r\n displayName: 'Swap staging and production slots'\r\n inputs:\r\n azureSubscription: '<Azure service connection>'\r\n appType: webAppLinux\r\n WebAppName: '<name of web app>'\r\n ResourceGroupName: '<name of resource group>'\r\n SourceSlot: staging\r\n SwapWithProduction: true\r\n - script: |\r\n curl -X PATCH -H \"Content-Type: application\/json\" -d '{\"maintenance\": false}' $(production_web_app_service_url)\/service-context\r\n name: deactivate_maintenance_mode_production\r\n displayName: Deactivate maintenance mode on production<\/code><\/pre>\nThe pipeline is triggered when a migration is pushed to the main branch.\nThe pipeline will then build the application,\nset the production and staging app in maintenance mode, apply the migration and\nswap the production and staging slots.<\/p>\n
Closing remarks<\/h2>\n
When planning a maintenance window for a database migration, it’s important to\nconsider the following:<\/p>\n
1) Identify the right time: Choose a time that has the least impact on end users, such as during off-peak hours or weekends.<\/p>\n
2) Communicate with stakeholders: Notify all stakeholders, including customers and internal teams, of the planned maintenance window well in advance. This will give them ample time to plan and prepare for the interruption in service.<\/p>\n
3) Test the migration process: Before the actual migration, test the process in a non-production environment to ensure it runs smoothly. This will also help identify any potential issues that need to be addressed.<\/p>\n
4) Have a rollback plan: In case something goes wrong during the migration, it’s important to have a rollback plan in place to quickly restore the previous version of the database.<\/p>\n
5) Monitor the migration: Continuously monitor the migration process to ensure it’s running smoothly and to quickly address any issues that may arise.<\/p>\n
6) By implementing a maintenance window strategy, organizations can minimize the impact of database migrations on end users and ensure the availability of the system during the migration process.<\/p>\n
It’s also important to note that some migrations such as those of very large databases or\nthose that require a lot of data transformation may require more than one maintenance window.\nIn such scenarios it is important to plan and test the migration in chunks, allowing for a more\ngradual and controlled migration process.<\/p>\n
Overall, a maintenance window strategy can be a useful tool for organizations looking to perform database migrations\nwhile minimizing disruption to end users. With careful planning, testing and monitoring, IT teams can ensure a\nsmooth and successful migration process.<\/p>\n
If you want to learn more or want to check out the code for yourself, you can find all source code here<\/a>. This will give a cookiecutter repository that provides you with a flask onion architecture with support for maintenance mode and tests with test containers<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Maintenance window implementation for database migrations using Flask.<\/p>\n","protected":false},"author":106000,"featured_media":14508,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[3367,178,3366,3368,3364,300],"class_list":["post-14494","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse","tag-database","tag-flask","tag-frameworks","tag-migrations","tag-open-source","tag-python"],"acf":[],"blog_post_summary":"
Maintenance window implementation for database migrations using Flask.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14494","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/106000"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=14494"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14494\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/14508"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=14494"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=14494"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=14494"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}