Skip to main content

Remote monitoring and alerting for IoT

How tools and practices used for monitoring cloud-native services apply to solutions that use IoT devices. Add operations visibility to remote locations.

Introduction

IoT devices produce many types of information, including telemetry, metadata, state, and commands and responses. Telemetry data from devices can be used in short operational timeframes or for longer-term analytics and model building.

Many devices support local monitoring in the form of a buzzer or an alarm panel on-premises. This type of monitoring is valuable, but has limited scope for in-depth or long-term analysis. This article instead discusses remote monitoring, which involves gathering and analyzing monitoring information from a remote location using cloud resources.

Operational and device performance data is often in the form of a time series, where each piece of information includes a time stamp. This data can be further enriched with dimensional labels (sometimes referred to as tags), such as labels that identify hardware revision, operating timezone, installation location, firmware version, and so on.

Time-series telemetry can be collected and used for monitoring. Monitoring in this context refers to using a suite of tools and processes that help detect, debug, and resolve problems that occur in systems while those systems are operating. Monitoring can also give you insight into the systems and help improve them.

The state of monitoring IT systems, including servers and services, has continuously improved. Monitoring tools and practices in the cloud-native world of microservices and Kubernetes are excellent at monitoring based on time-series metric data. These tools aren’t designed specifically for monitoring IoT devices or physical processes, but the constituent parts—labeled series of metrics, visualization, and alerts—all can apply to IoT monitoring.

What are you monitoring?

Monitoring begins with collecting data by instrumenting the system you’re working with. For some IoT scenarios, the system you’re monitoring might not be the devices themselves, but the environment and the process external to the device. In other scenarios, you might be interested in monitoring the performance health of the devices themselves, both individually and at the fleet level.

Consider the task of monitoring a human cyclist riding on a road. There are many different parts of the overall system you can monitor. Some might be internal to the system, such as the cyclist’s heart rate or sweating rate. Others might be external to the cyclist, such as a slope of the road, or external temperature and humidity. These internal and external monitoring goals can coexist. The methodologies and tools might overlap, but you can recognize these different domains—a physician might care about different measurements than the bike mechanic. Monitoring tools can be used to create custom monitoring views.

For example, you might organize your metrics into the categories that are discussed in this section. The specifics of how these are structured or combined will depend on the particular domain and applications.

Device hardware metrics

Device hardware metrics are measurements of the hardware or physical device itself, usually with some sort of built-in sensor. 

Firmware

Software running on the devices includes application software as well as the system software itself, which might be the operating system, or layers of a networking stack or device drivers. 

Application code

Application code on the device is specific to the role that device is performing in the system. 

External environment

Measuring the environment with sensors is often what people think about with regard to IoT devices. 

Cloud device interactions

An IoT solution is a complex system that includes software components that run both on the device and in the cloud. Understanding how these two systems interact requires you to understand what information each side has access to and how to bridge the two software runtime environments. 

Supporting systems

A complete monitoring solution requires monitoring both core and supporting components. Monitoring the application code on the device is an example of whitebox monitoring, where you’re interested in how the application is functioning. You probably also want to include some blackbox monitoring. For example, your monitoring software can probe APIs and other cloud services that your solution depends on. When you’re trying to respond to a problem, having these blackbox probes in place can lead to much faster resolution. 

Alerting

Alerting is about getting warnings or notifications, and helps draw your attention to important conditions. These in turn often lead you to check visualizations and often the associated log information.

A problem with alerting is that humans are good at learning to ignore annoying “noise” (think of traffic noise, repetitive emails, and so on). Alerts are only valuable if they can be responded to and then appropriately dismissed. If an alert reports an issue that can’t be addressed, the information in the alert should instead be another metric or visualization.

Source:

https://cloud.google.com/solutions/remote-monitoring-and-alerting-for-iot
https://cloud.google.com/solutions/iot-overview#operational_information
https://prometheus.io/docs/visualization/grafana/

Source Code Analysis Tools

Source code analysis tools, also referred to as Static Application Security Testing (SAST) Tools, are designed to analyze source code and/or compiled versions of code to help find security flaws.

Some tools are starting to move into the IDE. For the types of problems that can be detected during the software development phase itself, this is a powerful phase within the development life cycle to employ such tools, as it provides immediate feedback to the developer on issues they might be introducing into the code during code development itself. This immediate feedback is very useful, especially when compared to finding vulnerabilities much later in the development cycle.

Strengths and Weaknesses

Strengths
  • Scales well — can be run on lots of software, and can be run repeatedly (as with nightly builds or continuous integration)
  • Useful for things that such tools can automatically find with high confidence, such as buffer overflows, SQL Injection Flaws, and so forth
  • Output is good for developers — highlights the precise source files, line numbers, and even subsections of lines that are affected
Weaknesses
  • Many types of security vulnerabilities are difficult to find automatically, such as authentication problems, access control issues, insecure use of cryptography, etc. The current state of the art only allows such tools to automatically find a relatively small percentage of application security flaws. However, tools of this type are getting better.
  • High numbers of false positives.
  • Frequently can’t find configuration issues, since they are not represented in the code.
  • Difficult to ‘prove’ that an identified security issue is an actual vulnerability.
  • Many of these tools have difficulty analyzing code that can’t be compiled. Analysts frequently can’t compile code because they don’t have the right libraries, all the compilation instructions, all the code, etc.

Open Source or Free Tools Of This Type

  • Bandit – bandit is a comprehensive source vulnerability scanner for Python
  • Brakeman – Brakeman is an open source vulnerability scanner specifically designed for Ruby on Rails applications
  • Codesake Dawn – Codesake Dawn is an open source security source code analyzer designed for Sinatra, Padrino for Ruby on Rails applications. It also works on non-web applications written in Ruby
  • FindBugs – (Legacy – NOT Maintained – Use SpotBugs (see below) instead) – Find bugs (including a few security flaws) in Java programs
  • FindSecBugs – A security specific plugin for SpotBugs that significantly improves SpotBugs’s ability to find security vulnerabilities in Java programs. Works with the old FindBugs too,
  • Flawfinder Flawfinder – Scans C and C++
  • Google CodeSearchDiggity – Uses Google Code Search to identifies vulnerabilities in open source code projects hosted by Google Code, MS CodePlex, SourceForge, Github, and more. The tool comes with over 130 default searches that identify SQL injection, cross-site scripting (XSS), insecure remote and local file includes, hard-coded passwords, and much more. Essentially, Google CodeSearchDiggity provides a source code security analysis of nearly every single open source code project in existence – simultaneously.
  • Graudit – Scans multiple languages for various security flaws.
  • LGTM – A free for open source static analysis service that automatically monitors commits to publicly accessible code in: Bitbucket Cloud, GitHub, or GitLab. Supports C/C++, C#, COBOL (in beta), Java, JavaScript/TypeScript, Python
  • PMD – PMD scans Java source code and looks for potential code problems (this is a code quality tool that does not focus on security issues)
  • Progpilot – Progpilot is a static analyzer tool for PHP that detects security vulnerabilities such as XSS and SQL Injection.
  • PreFast (Microsoft) – PREfast is a static analysis tool that identifies defects in C/C++ programs. Last update 2006.
  • Puma Scan – Puma Scan is a .NET C# open source static source code analyzer that runs as an IDE plugin for Visual Studio and via MSBuild in CI pipelines.
  • .NET Security Guard – Roslyn analyzers that aim to help security audits on .NET applications. It will find SQL injections, LDAP injections, XXE, cryptography weakness, XSS and more.
  • RIPS – RIPS is a static source code analyzer for vulnerabilities in PHP web applications. Please see notes on the sourceforge.net site.
  • phpcs-security-audit – phpcs-security-audit is a set of PHP_CodeSniffer rules that finds flaws or weaknesses related to security in PHP and its popular CMS or frameworks. It currently has core PHP rules as well as Drupal 7 specific rules.
  • SonarQube – Scans source code for more than 20 languages for Bugs, Vulnerabilities, and Code Smells. SonarQube IDE plugins for Eclipse, Visual Studio, and IntelliJ provided by SonarLint.
  • SpotBugs – This is the active fork replacement for FindBugs, which is not maintained anymore.
  • VisualCodeGrepper (VCG) – Scans C/C++, C#, VB, PHP, Java, and PL/SQL for security issues and for comments which may indicate defective code. The config files can be used to carry out additional checks for banned functions or functions which commonly cause security issues.

GitLab has lashed a free SAST tool for a bunch of different languages natively into GitLab. So you might be able to use that, or at least identify a free SAST tool for the language you need from that list.

Source:
https://www.owasp.org/index.php/Source_Code_Analysis_Tools

Managed OS – Operating System Patching & Update Processes

Overview

Quality NOC provides reactive patching to servers with both standard and managed customer operating systems. For managed servers, patches and updates are available upon request to ensure no patches are applied without customer knowledge and consent, effectively reducing risks to application and data integrity. Data and/or reports are available for you in different ways, depending on the solution used.

OS Updates

On a monthly basis, Quality NOC Cloud engineers review all recommended patches made available by operating system software vendors. A subset of these patches and updates are identified by the OS vendor as Critical or Important to installed operating systems. The most recent version of our OS is given these updates and is then re-certified by Quality NOC before it is made available to be implemented with new server builds.

Managed Server Updates

As a part of making a server managed, more updates may automatically be applied to the server. Different Linux distributions and Windows are supported. Automatic updates can be pre-agreed with the customer and discussed in weekly status meetings.

Dedicated Server and Public clouds

Dedicated Servers and Public clouds platforms can be managed by our team of system administrator, Microsoft Windows servers and Linux distribution are supported.
We have deep experience installing new applications, scripts and applying system updates, or upgrades on Red Hat, CentOS, Ubuntu, Debian, and other linux distributions.
We can coordinate with your operations or devops team and schedule downtime, and do previous snapshots befoer patching or updating your OS or applications.

Extend your operations, DevOps or development team

We do weekly meetings with our customers, we can also do short daily meetings monday to fridays and continue chat on Slack/Wire/Skype, with your operations team, devops, or developers establishing a very fluid communication and reporting about progress and/or incidents.

All-in-One Monitoring Solution

 


We can use your own monitoring systems, we can install a new system for you and migrate the hosts or if you have a small operation you can use our own monitoring system.

Our NOC team help our customers with:

  • Website Monitoring
  • Application Performance Monitoring
  • Server Monitoring
  • Network Monitoring
  • Cloud Monitoring, Private and Public

We have customers of all sizes, from a large Telecom with more than 500 hosts, mid size and small companies with only 2 hosts. Our NOC team works around the clock 24x7x365.

Our customers save resources, their engineers have more time for important projects, we improve the operations and reduce downtime.

We have a custom solution for you.

Performance and customer satisfaction

This chart shows real customer data, the first month we handle all usual incidents.

Then started working proactively to reduce the number of incidents, improve the quality on detection of critical alarms, also working with thresholds and reducing downtime.

Clearly you can see how we reduced to 1/3 the amount of incidents. We continue working  the next months adding more services checks and gathering information to create reports.

We do a continue improvement of service checks, maintenance and proactive work to avoid incidents and increase up-time. After 6 month we probably manage to reduce to 1/4 the amount of monthly incidents and increase dramatically the up-time.

We are all very happy, our customers and us. We enjoy working with our customers.

Cool Dashboards with Grafana

Create good-looking dashboards for displaying time series data from Icinga2 with Graphite/Grafana

Monitoring your network and systems can be very cool using this nice tools

  1. Icinga2
  2. Graphite
  3. Grafana

This combination gives you the best from all worlds.

Precise monitoring using Icinga2, collect all data with Graphite and create good looking dashboards with Grafana.

Our NOC team can help you to get there and also take over all your system and network monitoring.