The Internet of Things is continuing to explode and its impact is being felt in every industry. With an expected value of over $1.6 trillion by 2025, it’s crucial for businesses to start exploring how the Internet of Things can benefit them. And that’s where IoT platforms such as AWS IoT Core come into play.
As founders, we always want to get our product to the market faster. If you have thought of an idea that will turn into a Million dollar product, you need to convert your idea into realization i.e get it developed — And Faster! Technology has evolved over a while. Nowadays, businesses can release products to the market much more quickly than in the past thanks to technological advances. Cloud technology, in particular, has made it simpler and faster to deploy applications. You can now access computing power, and all sorts of storage wherever you are, making it easier to get your product to market quickly. So if you are looking to shorten your product’s development cycle, look no further — Cloud is the key!
Similarly, if you have physical devices such as sensors and you want to exchange their data with other devices that may or may not be present at the same location as the sensor, Again — Cloud is the answer! You can install sensors on one side of the world which can easily communicate with devices installed in other parts of the world. This is the world of the “Internet of Things”.
Internet of Things can be defined as a system of computing devices that can exchange data over the internet without any kind of human interaction.
The trend of using Cloud services has led to a shift in the way products are developed. Years ago, when I started working on a product based on IoTs, we had to develop every service from scratch. We had to create services for Authentication, Authorization, Security, etc. But now, With Cloud services, we don’t need to develop every service. It’s often just a matter of configuring them on the cloud and integrating them with your application.
This blog will teach you all about AWS IoT Core, a service that enables you to quickly set up your IoT infrastructure. With AWS IoT Core, you don’t have to worry about almost anything related to IoT infrastructure. Be it the backend, or any process like authentication, security, etc — we will explore features provided by AWS IoT Core in-depth. Here is a the outline of this blog:
The most common architecture used in the Internet of Things is a Publisher-Subscriber or a Pub-Sub Model for exchanging data between different devices or nodes. In a Pub-Sub model, any device that generates or produces data is called Publishers. They can be sensor nodes such as temperature sensors, motion sensors, etc. These devices publish data to different topics such as HumiditySensor\Sensor1 . On the other hand, devices or nodes that want to access any kind of published data are called Subscribers. They can subscribe to any topic and start retrieving the data from that node. To make this possible, we use an MQTT Broker that is the heart of the Pub-Sub Model.
Keep in mind that all nodes publish the data to MQTT Broker which then sends the data to other nodes who have subscribed to that particular topic. Any node that has subscribed to particular topics needs to inform the broker. Once the node subscribes, it starts receiving data that is published on that topic. Keep in mind that if the MQTT broker is down, then the complete communication will be down and nodes will not be able to exchange data.
MQTT stands for “Message Queuing Telemetry Transport” that is a lightweight protocol which can also be used by Microcontroller devices. Microcontrollers usually have very limited resources making it a good fit. The protocol is bi-directional, scalable to millions of devices, supports reliable message delivery in unreliable channels, and provides security.
MQTT Broker is an application you can install on your computer. Devices can connect with the MQTT broker using the MQTT protocol. “Mosquito MQTT Broker” is a very common service that you can install on your machine. You can follow this link if you want to install and give it a try. To communicate with the server, many libraries are available (based on your favorite programming language) that implement the MQTT protocol. For example, if you are a Python lover, you can install “paho-mqtt” and start communicating with the broker. Here is an example of the python code that will connect to the MQTT Broker and then start receiving data from some other node publishing data at HumiditySensor\Sensor1 .
import paho.mqtt.client as mqtt
# This function is called when this machine or node can successfully connect with the MQTT Broker.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc)) # Subscribe to a topic where some other node would be publishing
client.subscribe("HumiditySensor\Sensor1")
# This function is called whenever data is published on the topic this device has subscribed
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))# Configure the MQTT Client
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message# Assuming you have configured your MQTT Broker at 192.168.0.13 on your local machine
# 1883 --> MQTT Port
# 60 --> Timeout Interval
client.connect("192.168.0.13", 1883, 60)# Keep the server running forever
client.loop_forever()
Before you get too excited about implementing your million dollar IoT idea using MQTT broker as explained above, there are a number of things you need to take into account.
All problems have solution. Your team can sit together and come up with a way to solve all the listed problems. It will require your time as well as money. Wouldn’t it be awesome if I tell you that AWS IoT Core can handle all the things for you? that is why we are going to explore this amazing service from AWS. In Part-I of this blog, we will explore how can we provision the nodes, how can we manage the nodes.
All the problems list above have a solution. Your team can sit down and come up with a way to solve every single problem without any doubt. However, it will required you to put a lot of effort and money which is not what you want — Right? That’s why we’re going to explore AWS IoT Core which is an amazing service from AWS.
According to the landing page of AWS IoT Core — “AWS IoT Core lets you connect billions of IoT devices and route trillions of messages to AWS services without managing the infrastructure”. This is a concept also called “Serverless”.
Serverless: If you don’t have to manage the underlying infrastructure (or resources), we call it Serverless computing. It doesn’t mean that there are no servers. It just means that someone else is responsible to manage that for you and you don’t have to bear the pain.
AWS IoT Core provides you features that can quickly help you to implement complex functionality across your IoT Network.
Note: A Device, Node, and Thing are essentially the same thing in the context of this blog. AWS IoT Core calls every IoT Device a “Thing”. We will be using these words interchangeably throughout the blog.
To access AWS IoT, you need to log in to your AWS Account. If you are new to AWS, you can register a new account on AWS for free. However, you need a credit card to register. Once you register, you get a 12-month free-tier account accessing most of the services for free.
After logging in to your account, can search “IoT Core” in the search bar. You can then click on “IoT Core”.
This will lead to the AWS IoT Core Console screen as shown here. On the left navigation bar, different options are grouped based on functionality.
AWS IoT Core is essentially an MQTT Broker with additional features on top. As we discussed MQTT Broker previously, AWS IoT Core provides a managed MQTT Broker that provides low latency and high throughput. You don’t even have to manage the installation, maintenance, or security of this broker. If your device is authorized, it can securely transmit or receive messages. You don’t have to worry about the scalability of your IoT Network. You can increase the number of IoT devices without worrying to increase the capacity of the underlying resources as it is managed by AWS automatically. Here is an overview of different features provided by AWS IoT Core:
AWS Lambda Functions: According to AWS, “Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources”
Important: Based on the AWS region you select, some features might not be available. You can always change the region in AWS from the top right of the console.
AWS IoT Core provides a very user-friendly dashboard that provides some basic IoT metrics. You can access it by clicking “Monitor” on the left navigation bar providing stats like successful connections, Rules executed and various other metrics which can help you identify problems in your IoT Network.
Let us start configuring our first nodes using wizards. These wizards hide all the additional options that are required to provision a node from the user. You can provision either a single device or multiple devices at the same time. If you expand “Connect” in the navigation bar and click “Get Started”, you will be provided with two options.
Let us see try to onboard a single device. Click the “Get started” button under the “Onboard a device”. You will see the instruction screen. You can read them if you want. Click the “Get Started” button on the bottom right of the screen to continue.Let us see try to onboard a single device. Click the “Get started” button under the “Onboard a device”. You will see the instruction screen. You can read them if you want. Click the “Get Started” button on the bottom right of the screen to continue.
On the next screen, you can select the platform and the SDK. These options will generate a script at end that will help you to configure and install the required libraries. If you are using Microcontrollers, you can select any combination and click “Next”..
The next screen will provide you different configurations to identify your node. The most important is the name of your device. For now, let us call it “MyFirstThingName”.
By default, some optional configurations will be hidden. You can expand those by clicking on “Show optional configuration”. These options can also be modified after completing this process. Thus you can safely ignore these for now.
Click “Next Step”. After a few seconds, you will see the following screen indicating that you have successfully provisioned a node.
As you can see on the screen, it has automatically created a policy document called “MyFirstThingName-Policy” which provides permission to your node to connect to the MQTT Broker and publish/subscribe to allowed topics in the policy. If you want to know more about policy document, jump to this section “Policy Statement — Device Authorization” of this blog.
It also generated certificate, and public and private key that is required for authentication and secure communication as well. You additionally get a script file that can be executed in the OS you selected which will install the dependencies required to communicate with AWS IoT Core. Download the connect kit into your machine. It is a ZIP file that can be extracted into any folder. The content looks like this:
There are four files:
Note: The AmazonRootCA certificate is required to establish secure and encrypted communication with AWS in general. It has nothing to do with AWS IoT Core.
Once you download the file, the “Next Step” button will be accessible which will take you to the instruction screen. Click “Done”.
Congratulations, you have successfully provisioned your first IoT device with merely a few clicks.
If you build your own IoT infrastructure, authenticating and securing devices can be quite a challenge. As you saw, provisioning devices is extremely easy in AWS IoT Core. Devices that are not registered or provisioned with IoT core are not able to communicate with AWS IoT Core.
If you have hundred or even million devices, provisioning each device individually can be a pain and time-consuming. Primarily AWS IoT Core provides two ways to provision multiple devices:
To access Bulk Registration screen, you can click “Activity” → “Start registration”
Let us discuss all the fields one by one.
1805e1f631a11fa4097d9307ec103d343111902ef29ba3e3fa00e63d570fa47a">
294cd70210ab6a1000069882d1f8ba33b8075825176569770031b5b9c329b283">
A Note on CertificateID: You will need to generate the certificates in your AWS IoT Core and add their IDs in the above file. To generate certificates from AWS IoT Core, click “Secure” on the left navigation bar and select “Certificates”. You can further click “Add certificate” to manually generate the certificate. If you want to follow along, you need to generate two certificates and paste their certificateID in the above JSON file.
Now that we have all the pre-requisites, you need to tell AWS IoT Core how to use the JSON template to provision new devices. The “Ref” keyword is used to reference the values in the JSON template in the provisioning template. The template will be used by AWS to provision nodes, attaches a certificate (used for authentication) and the provided policy (used for authorization) automatically. Here is a the provisioning template that we are going to use in this example:
"Parameters" : "ThingName" : "Type" : "String"
>,
"SerialNumber" : "Type" : "String"
>,
"Location" : "Type" : "String",
"Default" : "LA"
>,
"CertificateId" : "Type" : "String"
>
>,
"Resources" : "thing" : "Type" : "AWS::IoT::Thing",
"Properties" : "ThingName" : ,
"AttributePayload" : < "version" : "v1", "serialNumber" : >
>,
"OverrideSettings" : "AttributePayload" : "MERGE",
"ThingTypeName" : "REPLACE",
"ThingGroups" : "DO_NOTHING"
>
>,
"certificate" : "Type" : "AWS::IoT::Certificate",
"Properties" : "CertificateId":
>
>,
"policy" : "Type" : "AWS::IoT::Policy",
"Properties" : "PolicyDocument" : " < \"Version\": \"2012-10-17\", \"Statement\": [< \"Effect\": \"Allow\", \"Action\":[\"iot:Publish\"], \"Resource\": [\"arn:aws:iot:: :topic/foo/bar\"] >] >"
>
>
>
>
Important: Please replace and with your region and AccountID. If you are not sure, you can go to Secure → Policy. Select any policy. Under the details → policy ARN, you can see your region and your account ID.
The final configuration looks like in the following screen. Click “Start registration” once you are done.
If you have entered all the information correctly, the provisioning process will start. It might take sometime based on the number of nodes you are going to provision. Since we are provisioning only two nodes, it should be almost immediate. You can always go back to “Activity” page to see the status of the bulk registration. In case there are any errors, you can select your TaskID, click Action → Failure log. This will download a text file containing the failure reason. As you can see the latest TaskID contains two success count which is the number of nodes we want to provision, hence we are successful.
To confirm, you can go to Manage →Things and confirm if “foo”and “bar” have been successfully created.
If you have a specific requirement and you want to configure each manually, then you need to go to the Manage → Things and click on the “Create thing” button. It is similar to the process you followed earlier. However, it provides you with more and flexible configuration options. This is for more advanced users.
The next screen will provide you two option i.e to either create a single thing or multiple things. The “Create many things” will lead you to the “Bulk Registration” option which we have already explored.
Let us proceed with “Create Single thing”. Enter the name of your new device. Let us call it “MySecondDevice”. Leave the rest of the options as default.
Scroll down where you see Device Shadow. We will leave it with “No Shadow” for now as we are simply exploring the features. You can always attach shadow with your device later. Click “Next” to proceed.
Shadow: Device Shadows are similar to databases that are specific to each device. You can have a single shadow or multiple shadows as required by your application
In the next screen, we need to configure our certificates. Here you have the option of either choosing AWS to generate the certificates for you, using your certificates, or even can register your own Certificate Authority and generate your certificate. This is a costly process and I would suggest you let AWS auto-generate a new certificate for you. You can always create a certificate later and attach it with your Thing. Click “Next” to proceed.
Certificates: Certificates are used to authenticate a device and thus allow it to be considered an authentic device in the IoT Infrastructure. In case, your IoT device is compromised, you can revoke the certificate thus preventing the device to communicate with AWS IoT Core
In the next screen, you need to attach a policy with your device. If you want to create a new Policy, you can press the “Create policy” button. Previously, the wizard automatically generated the policy document for us, we will use the same policy for our new device. Select the “MyFirstThingName-Policy” by click the checkbox on the left side of it and then click “Create thing”.
A new popup will appear. You need to download all the certificates provided into a secure location on your computer.
Important: The public and private keys are only downloadable on this screen. In case you lose any of these keys, you will have to provision a new device from scratch.
Once you downloaded all the certificates and the keys, press the “Done” button. You have now successfully created a new device in a manual fashion.
If you have followed along, you should now see 4 devices on the screen.
While we provision devices in AWS IoT Core, we attach a policy document “Policy” with every Thing or node that is being provisioned. A Policy document is a simply a JSON structured file that you attach to any device which authorizes it to do the following operations:
Here is an example policy. In this example policy, the node can only publish, subscrive and recieve data from $/room*
"Version": "2012-10-17",
"Statement": [
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:us-east-1:123456789012:client/$"
]
>,
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:us-east-1:123456789012:topicfilter/$/room*"
]
>,
"Effect": "Allow",
"Action": [
"iot:Receive"
],
"Resource": [
"arn:aws:iot:us-east-1:123456789012:topic/$/room*"
]
>,
"Effect": "Allow",
"Action": [
"iot:Publish"
],
"Resource": [
"arn:aws:iot:us-east-1:123456789012:topic/$/room*"
]
>]
>
Wildcard characters: AWS IoT Core policies use * and ? as wildcard characters. In a policy document, the * represents any combination of characters and a question mark ? represents any single character.
To view the policies that we have already created, go to Secure → Policies. If you have followed along, “MyFirstThingName-Policy” is the one created while provisioning a single thing using the wizard. The second policy is the one created by the Bulk Registration process.
Click on “MyFirstThingName-Policy”. In the next screen, click on “JSON” toggle button to view the policy document in JSON format. Here you can modify the policy and publish new versions of the policy document. This allows you to revert to a previous version if something goes wrong.
If you haven’t modified the The policy document for “MyFirstThingName-Policy”, then it looks like the following:
"Version": "2012-10-17",
"Statement": [
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive",
"iot:RetainPublish"
],
"Resource": [
"arn:aws:iot:us-east-2:906961234567:topic/sdk/test/java",
"arn:aws:iot:us-east-2:906961234567:topic/sdk/test/Python",
"arn:aws:iot:us-east-2:906961234567:topic/topic_1",
"arn:aws:iot:us-east-2:906961234567:topic/topic_2"
]
>,
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:us-east-2:906961234567:topicfilter/sdk/test/java",
"arn:aws:iot:us-east-2:906961234567:topicfilter/sdk/test/Python",
"arn:aws:iot:us-east-2:906961234567:topicfilter/topic_1",
"arn:aws:iot:us-east-2:906961234567:topicfilter/topic_2"
]
>,
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:us-east-2:906961234567:client/sdk-java",
"arn:aws:iot:us-east-2:906961234567:client/basicPubSub",
"arn:aws:iot:us-east-2:906961234567:client/sdk-nodejs-*"
]
>
]
>
The summary of above policy is as follows:
Note: If you have used a different region that you might not see “us-east-2”. It would be according to what region you have selected in your AWS Console.
Using AWS IoT Core policies, you can create a different policy for every node or you can share the same policy with multiple nodes depending upon the need of your application. This is an extremely useful feature and the console lets you manage these policies with ease.
One of the very useful features of AWS IoT Core is the Device Shadow. Shadow is a kind of database that is specific to each separate node. It can be used to store the latest (reported) state of the device which can be read or accessed by any other device in the infrastructure whenever required. So even if the node goes offline, other nodes can still access the reported states from its shadow. The information in the shadow is stored in form of “Desired” and “Reported” states. If some device wants to change the state of a particular node, it sends the request by adding key-value pair in the desired state in the shadow. Whenever there is a change in shadow document, the corresponding IoT node, reads the desired information from its shadow, process the information and sends the updates in the reported state. The device that initially requested the data gets an update that the reported state has been updated. This data is persistent and stays as long as it is manually deleted or by some process. Here is a typical example of how data is stored in a shadow.
"state": "desired": "attribute1": integer2,
"attribute2": "string2",
.
"attributeN": boolean2
>,
"reported": "attribute1": integer1,
"attribute2": "string1",
.
"attributeN": boolean1
>
>,
"clientToken": "token",
"version": version
>
If you wish to add a shadow into any “Thing”, go to Manage → Things and select any device that you have already provisioned. Click on “Device Shadow”. Since we created “MySecondDevice” without this service, there are no existing shadows found.
Click on “Create Shadow” to add a new one. You will see a popup with two options:
Select “Unnamed (classic) Shadow” and click “Create”.
It also generated the MQTT topic $aws/things/MySecondDevice/shadow for any device to communicate with this shadow service of the “MySecondDevice” node. i.e any device if allowed by policy statement can publish or subscribe to this shadow. If you want to learn more about the Shadow service, please see the following link for APIs and details. https://docs.aws.amazon.com/iot/latest/developerguide/iot-device-shadows.html
AWS IoT core provides a number of features that allows the development team to quickly deploy the IoT infrastructure for their product. Instead of worrying about the infrastructure, its security and management, just focus on the core business logic of the product. In this blog, we were able to discuss the core features like provisioning nodes, bulk registrations, configuring policies, shadows etc.
According to the pricing page of AWS IoT Core, If you have 100,000 devices that maintain a constant connection to AWS IoT Core for 30 days in the Europe (Ireland) region — Each day, each device sends 325 messages of 1 KB in size. Of the 325 messages sent per device each day, 100 trigger a Device Shadow update and 200 trigger a rule that executes one action, your total cost would be around $1,876.60 per month which is very reasonable.
With all the benefits listed in this blog, If you are a CEO or CTO of any tech company, you should definitely consider AWS IoT Core for your next million dollar product.