Announcing Azure Cosmos DB custom routing endpoints for Azure IoT Hub – GA
We are very happy to announce the general availability for Azure Cosmos DB custom routing endpoints for Azure IoT Hub.
Many IoT solutions require extensive downstream data analysis. You have to seamlessly push the data into hyperscale databases quickly. For example, intelligent transport systems have a constant inflow of a continuous stream of data. This data must be analyzed even as it is being written into the database.
We have seen multiple scenarios in manufacturing, automotive, and other IoT solution-focused industries that require hyperscale databases. These solutions need extremely high throughput to process the continuous stream of data. Traditional SQL-based relational databases cannot scale optimally and also become very expensive once data scale increases. Azure Cosmos DB is designed to scale horizontally, allowing it to ingest vast amounts of streaming data in parallel while maintaining very low write latency and high availability. These attributes make it best suited for such scenarios.
Until today, you had no direct way to get IoT device data from Azure IoT Hub into Azure Cosmos DB for analysis. You had to either set up a custom application or use other Azure services like Azure Functions or Azure Stream Analytics to collect the streaming data and then write to Azure Cosmos DB. This not only meant additional development effort from your team but also increased the overall cost of your solution. But you don’t need to do this anymore!
Azure IoT Hub now supports the ability to send device data directly into Azure Cosmos DB. You can
- Directly configure an Azure Cosmos DB account as a custom message routing endpoint within Azure IoT Hub.
- Use synthetic Partition Keys to auto-generate partition keys based on your business logic.
- Create routing rules to redirect specific data to the configured Cosmos DB account.
How can I configure the Azure Cosmos DB account within my Azure IoT hub?
You can configure the Azure Cosmos DB account within Azure IoT Hub quickly. In the Azure portal, you first navigate to the Azure IoT Hub you have set up. In the Settings pane, you select “Message Routing” and click on “Custom endpoints” and choose “Cosmos DB” from the dropdown list. Once you select Cosmos DB as your preferred endpoint, the below screen appears – you configure the endpoint and click on “Create” to complete the process. You have successfully completed the route setup to Azure Cosmos DB from Azure IoT Hub.
Can I use Synthetic Partition Keys?
Yes, you can. You can use the Azure IoT hub to write to Azure Cosmos DB in JSON (if specified in the message content type) or as a Base 64 encoded binary. You can also enable Synthetic Partition Keys to create logical partitions of your data automatically based on your business logic and data growth rate.
Azure Cosmos DB is horizontally scalable database. All data/documents written to it must contain a property that represents a partition key that distributes the data across the underlying physical partitions. You define the partition key property name at the Container level. You cannot change it once it has been set. Each logical partition has a maximum size of 20GB. To effectively support high-scale scenarios, you can enable Synthetic Partition Keys for the Azure Cosmos DB endpoint and configure them based on your estimated data volume. For example, in manufacturing scenarios, your logical partition may approach its max limit of 20 GB within a month. In that case, you define a Synthetic Partition Key which is a combination of the device ID and the month. Each new Cosmos DB record will automatically add this key to the partition key field. You have now ensured logical partitions are created each month for each device.
What next?
Setting up of Azure Cosmos DB endpoint from Azure IoT Hub is generally available from 5th October 2023. You can learn more about the functionality from our documentation. You can also reach out to us via various support tools if you have any questions.
About Azure Cosmos DB
Azure Cosmos DB is a fully managed and serverless distributed database for modern app development, with SLA-backed speed and availability, automatic and instant scalability, and support for open-source PostgreSQL, MongoDB, and Apache Cassandra. Try Azure Cosmos DB for free here. To stay in the loop on Azure Cosmos DB updates, follow us on Twitter, YouTube, and LinkedIn.
Light
Dark
0 comments