EyeShield : Simple PoC of Shoulder Surfing Attack Detector.

There is an interesting paper titled “Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing Attacks” from USENIX conference, this paper have a nice approach to handle Shoulder Surfing Attack.

The paper introduces a novel approach to mitigate shoulder surfing attacks on mobile devices. Recognizing the increasing reliance on mobile devices for sensitive tasks, the authors develop a real-time protection mechanism that leverages the device’s front camera to detect potential attackers peeking over the user’s shoulder. Using advanced image processing and machine learning algorithms, the system identifies and classifies onlookers, determining whether they pose a threat. When a threat is detected, the system dynamically alters the display or alerts the user, ensuring screen confidentiality. Through rigorous evaluations, the approach demonstrates high accuracy and minimal performance overhead, promising a practical solution to a growing security concern.

Shoulder Surfing Attack

So based on that paper I build a Proof of Concept to replicate this attack detection, I’m using Golang and GoCV to build the PoC. In this PoC using Haar Cascade Algoritm to detect multiple face using primary camera and display popup notification to notify “Possibility Shoulder Surfing Attack Detected” or there is a possibility to blur all the screen.

This PoC using this code :

package main

import (
 "fmt"
 "gocv.io/x/gocv"
)

func main() {
 windowOverlay := gocv.NewWindow("Overlay")
 defer windowOverlay.Close()
 img2 := gocv.IMRead("hacker.jpg", gocv.IMReadAnyColor)

 // parse args
 deviceID := 0
 xmlFile := "haarcascade_frontalface_default.xml"

 // open webcam
 webcam, err := gocv.OpenVideoCapture(deviceID)
 if err != nil {
  fmt.Printf("error opening video capture device: %v\n", deviceID)
  return
 }
 defer webcam.Close()

 // open display window
 window := gocv.NewWindow("Face Blur")
 defer window.Close()

 // prepare image matrix
 img := gocv.NewMat()
 defer img.Close()

 // load classifier to recognize faces
 classifier := gocv.NewCascadeClassifier()
 defer classifier.Close()

 if !classifier.Load(xmlFile) {
  fmt.Printf("Error reading cascade file: %v\n", xmlFile)
  return
 }

 fmt.Printf("Start reading device: %v\n", deviceID)
 for {
  if ok := webcam.Read(&img); !ok {
   fmt.Printf("Device closed: %v\n", deviceID)
   return
  }
  if img.Empty() {
   continue
  }

  // detect faces
  rects := classifier.DetectMultiScale(img)
  fmt.Printf("found %d faces\n", len(rects))

  if len(rects) > 1 {
   fmt.Printf("Possibility Shoulder Surfing Detected!!\n")
   windowOverlay.IMShow(img2)
   windowOverlay.SetWindowProperty(gocv.WindowPropertyFullscreen, gocv.WindowFullscreen)

  } else {

   windowOverlay.Close()
  }

  // show the image in the window, and wait 1 millisecond
  window.IMShow(img)
  if window.WaitKey(1) >= 0 {
   break
  }
 }
}

Result :

Face Detection

When detect multiple faces its possible a shoulder surfing attack

Let me know if you need any further information or details from the paper!