Daily push notifications using R, AWS and Pushover

July 05, 2017 | 4 Minute Read

My girlfriend and I are in the midst of buying a house and therefore we need to take out a mortage - preferably at the best possible price. So I want to keep myself updated on the current price of the mortage we are eyeing in order to strike when the price is just right.

To my surprise I was unsuccesful in finding any kind of service that would shoot a daily notification to me containing the current mortage price (Well, I wanted to test out a solution in R, so i didn't bother looking THAT hard for suitable service). So that's what I'm trying to achieve and describe in this post!

The tools needed are:

• R and Rstudio
• Amazon Web Services (AWS)
• MobaXterm (or another SSH client)
• Pushover.net API

I have Rstudio set up on a EC2 instance from AWS - I won't go over how to install Rstudio on a AWS instance, I followed this and this tutorial and I suggest you do the same. The next step is to write a simple R script that scrapes a website for mortage prices and uses Pushovers API to send out a notification. MobaXterm is used to remotely access the AWS instance and set a cron job to run the R script daily.

The mortage price is scraped from Sydbanks website and I use SelectorGadget (a Google Chrome add-in) to identify precisely the CSS selector of interest. In this case it is the current sales price (td:nth-child(3)) and the point-change (td:nth-child(7)) since yesterday. The CSS selector are then fed to the html_node() and html_text() functions from the rvest package which graps the price and point-change and store them as a string.

  #################################### BASICS #############################################

  # Author: Mads Bønding
  # Date:05-07-2017
  # Document version: 1.0
  # R version: 3.4.0 sessionInfo()
  # R Studio: 1.0.143
  # Purpose: Scrape mortage prices and send a daily push notification with current price to my phone/watch using R, AWS and Pushover.


  ################################## LIBRARIES ############################################
  library(dplyr) # v. 0.4.3
  library(rvest) # v. 0.3.2
  library(notifyR) # v. 1.02.0

  ################################# DATA WRANGLING ###########################################

  # read and store the url
  mortage_prices <- read_html("http://www.sydbank.dk/investering/analyser/obligationer/obligation?kode=DK0009504755&type=Obligation&sektor=NONE&straks=true")

  # Extract the relevent information from mortage_prices
  current_price <- mortage_prices %>%
    html_node("td:nth-child(3)") %>%
    html_text()

  daily_change <- mortage_prices %>%
    html_node("td:nth-child(7)") %>%
    html_text()

Next step is putting together a message to send and actually sending it. I am using the send_push() function from Torben Engelmeyers notifyR package to send myself a push notification. NotifyR uses the API from Pushover and I need to create an account on pushover.net and copy the userkey to the send_push() function. I also need to download the Pushover app to my phone in order to receive the notifications. Then use the paste() function to construct the message containing current_price and daily_change and include the message in send_push().

  userkey <- 'xxxxxxxxxxxxxxxxxxxxxxxxxx'

  message <- paste("The price on 2.pct without repayment is right now ",current_price, " and has changed ", daily_change, " since yesterday.")

  send_push(userkey, message, title = "Mortage price")

With the script sorted out the next step is to make it run daily on my AWS instance. For windows users there is neat little addin to Rstudio that easily lets you schedule your R scripts but since my EC2 instance is running an ubuntu server I can't use this method.

I need to start a SSH session in MobaXterm(or any other SSH client) and connect to the AWS instance and then run the command crontab -e. This will open an editor that lets me specify when and which script I want to schedule. All I need to do is add the line pictured below to the editor which specifies the path to the script I want to schedule and that I want it to run daily at 7.15 a.m (be aware that the script will execute based on the timestamp of the server which may not match your local time). Hit exit and save the file and the script is scheduled!



It works! I now get daily updates on the mortage price. As a bonus I got to try out some new things and killed a couple of hours putting it together...