AWS Timestream is a managed time-series database. This makes it easy to store and analyze trillions of events per day up to 1,000 times faster and at as little as 1/10th the cost of relational databases. It is designed for IOT, Devops or general analytics use cases where you have large amounts of data for various time intervals and you need a way to store it without the overhead of server management. To help show the power of it I have created an example here where if you have an iPhone it will take the activity data from the Health app and send it into Timestream with an AWS Lambda function. We can then visualize all the data with Grafana to view detailed trends in our activity. All the infrastructure will be created by AWS CDK to leverage the benefits of infrastructure as code (IAC). Xerris has other post’s on CDK if you want to hear more of the benefits of that tool.
Here is a quick visual overview of the architecture we are going to build:
Timestream Lambda Handler
We will start off with the Lambda Node.JS function code which will take in the data and push it into Timestream. Note that you will need to package a newer version of the AWS SDK in your node modules because the version included with lambda does not yet support Timestream.
First the Timestream client “writeClient” is created and then we have a route called for the path /healthInput that takes all the records and inserts them as measures in Timestream. Measures are the primary value you want to track per record and in this case we are tracking the step count we took. The other way to track additional data is through Dimensions. In this case the only metadata we are including is the duration which we are setting to hourly. So each entry will represent an hours worth of steps.
After that we just tell the Timestream client which database and table we want to write to and that covers the extent of this function. You can add a ton of software engineering depth to this but I wanted to focus primarily on writing to Timestream and to show what is possible.
Infrastructure Setup with CDK
Now let’s spin up the AWS Architecture with CDK written in C#. This covers the creation of the lambda, the API Gateway in front of it as well as the Timestream database itself. For any further details on this you can see other Xerris blog posts on CDK.
The only thing I want to mention about Timestream setup is about the memory store and the magnetic store. The memory store is a more expensive but faster storage tier and the magnetic store is a cheaper but slower storage tier. When creating the table you are required to give retention periods for both tiers. So when you send data to timestream it starts off in the memory store and then after that retention period ends it goes to the magnetic store. Then finally when the magnetic store retention period ends the data is deleted.
So you need to make sure that any data you are adding to Timestream has a timestamp that is within the memory store retention period because all data needs to start there. Another thing to note that currently in CDK there is no way to set either of these values so if you want to update the defaults you need to manually update them. This will definitely be updated as time goes on but Timestream is still a new service so this is to be expeted.
Sending the activity data using HealthKit on iOS
Now let’s go and create a quick iOS app that will send the data to our Lambda. All the code is availiable but I won’t go into too much detail. All you need to know is that we authorize the app to access Healthkit and then pull the step data for the last month. Then we send that off to the lambda we created. You will need to update the url of the request on line 58 with the default domain of the API Gateway that was created for. In a production iOS app you would use packages like Alamofire for the API requests but for this I am using pure Swift and iOS API’s to keep everything simple. This also includes the code for the View which is written in SwiftUI.
When you run the app on your iPhone you will see it pulls and shows your steps for the day and then in the background it pushes your data to the lambda that will populate Timestream.
Now let’s open up the AWS console and explore the Timestream Query editor to see how our data is being populated. Timestream can be fully queried by standard SQL queries so here we are doing a SELECT * of the table but limiting it to 10 results. We see the data that was from our iPhone:
Visualizing with Grafana
Now we can start to visualize the data we have created in Timestream. For this I am using Grafana which is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. We are going to be connecting it to Timestream here to visualize our activity data.
First go and install Grafana on your machine from here and log into the admin user. You will also need to install the Timestream plugin from here. We are going to go to Configuration -> Data Sources and add our Timestream Table. Setup the auth provider, region, and endpoint settings and now we can start playing with the data. Go to the Dashboards area and create a new Graph visualization. For the query we are going to set it to pull all of the data in our table:
SELECT * FROM $__database.$__table
Now go and play with the data and see the times of day you are most active!
Overall while Timestream is a new service I found it very easy to use and because there are no servers to manage and no capacity to provision, you can just focus on building your applications. It provides high throughput ingestion, rapid point-in-time queries through its memory store, and fast analytical queries through its cost optimized magnetic store. Then as well, you pay only for the data you ingest, store, and query.
Thank you for reading and you can find the full code here. If you want to find out more about Timestream and other AWS services feel free to get in touch with us at Xerris and we can help you craft innovative cloud focused solutions for your business.