Auto-raise support hours as morning alarm with Rust

At iLert we are big friends of dogfooding as we love to take our features and check out any way possible to hack or integrate them.

We have launched the auto-raise feature for support hours in early 2020 and they really are a fancy way to setup wake-up calls in the morning.

Explaining the terms first

In case you already know iLert, you can skip this part and start right at The wake-up call.

What are “alert sources” again?

Alert sources are the basis of any kind of interaction in iLert. An alert source is a self-service inbound connection to your preferred monitoring software or other tool.

What are “support hours” again ?

Support hours help you manage the priority of incoming alerts for an alert source depending on the current time and day. E.g. from Monday to Friday between 8 am and 5 pm you want to be immediately alerted and outside of these hours the alerts should not keep you from enjoying your free time.

The “auto-raise feature?”

But what if you want to be immediately informed in case there have been any alerts during your well deserved free time after coming back into the office on Monday 8 am?

We got you covered with the auto-raise feature for support hours. All you have to do is enable the “Raise priority” checkbox and we will raise the incident’s priority for the alert that came up during your off-time as soon as your next support hour start.

The wake-up call

We have multiple small check applications in different zones across cloud providers to validate the response and uptime of our service. These small apps are actually Rust binaries which constantly poll our API and run through test operations e.g. create and update an escalation policy.

In case any of these checks fail they create an incident right in iLert - but additionally once a day they create a sum-up incident with the response statistics of the last 24 hours.

We are going to use this daily sum-up event to wake us up in the morning with a call informing us about the latest statistics to start the day right.

Setting up an alert source for the sum-up event

• Create an account (if you have not already) at iLert for free
• Make sure your user (Click on your name in the top right then on ‘My Profile’) has configured a valid phone number
• And voice calls enabled in the high priority notification settings
• Navigate to ‘Alert sources’ and create a new alert source:

We want the new alert source to be of integration type: ‘API’ with incident creation enabled.

Choosing ‘High during support hours’ helps us to receive our sum-up updates all day in iLert (except for our active time from 8 - 9 am) with a low setting (only sending us an email per default). But once we reach a new support hour window at 8 am, the event will be risen and we will receive a voice call with its details (as configured in our profile above).

Sending events to the iLert API with Rust

We are using the iLert Rust client to send the event to iLert’s API. Make sure to replace the placerholder with the API key exposed by your alert source.

use ilert::ilert::ILert;
use ilert::ilert_builders::{UserApiResource, EventApiResource, ILertEventType};

let mut client = ILert::new().unwrap();

let as_api_key = "YOUR-ALERT-SOURCE-API-KEY-HERE";
let event_result = client
    .post()
    .event(
        as_api_key,
        ILertEventType::ALERT,
        "These are our sum-up statistics: Everything is fine.",
        None,
        None)
    .execute()
    .unwrap();

‍

You can also find the API documentation here for your client or simply try out the curl command below :)

curl --location --request POST 'https://api.ilert.com/api/v1/events' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
	"eventType": "ALERT",
	"apiKey": "YOUR-ALERT-SOURCE-API-KEY-HERE",
  "subject": "These are our sum-up statistics: Everything is fine."
}'

Now we just have to make sure that the event is triggered every 24 hours. (The Rust excerpt below is just a mockup, I would not recommend using this in production environments.)

let mut last_run = Instant::now();
loop {
    thread::sleep(Duration::new(0, 25000000));

    if last_run.elapsed().as_secs() >= 86400 {
        last_run = Instant::now();
        sendSumUpEvent();
    }
}

When the event is received outside of the support window, a low priority incident is created.

But as the support hour window is approached it will be automatically risen and become a high priority incident, which will cause an outbound voice call that wakes us up with the incident summary in the morning.

After listening to the wake up call we can straight up answer by pressing ‘1’ on our phone to resolve the incident.

Conclusion

Surely in 99% of the cases you would only ever sent events that actually evolved out of some kind of issue or problem of which you want to be informed about.

However it is always good feeling to start the day knowing that everything ran smoothly while you were asleep.

‍