Hard hat wearing detection based on head keypoint localization
person
head
person
head
person
head
person
head
hard hat
hard hat
no hard hat
hard hat on
no hard hat
hard hat on
Input image
Detected hard hats
Detected people with
localized head keypoint
Head keypoint inside
hard hat bounding box
No hard hat in place of
head keypoint
Rule based hard hat wearing classification
Figure 1: Workflow of the proposed solution.
dependent. On the other hand, Shen et al. (2021) solution
works only when a person face is at least partially visible.
3. Keypoint based hard hat wearing detection
3.1. Proposed solution
Most of the solutions used to detect the wearing of the
hard hats presented in Section 2 fall into one of the two gen-
eral categories:
• detection of people or people and hard hats, wearing
a head protection is determined in different steps ac-
cording to rules or another models,
• detecting hard hat wearers and non-wearers as sepa-
rate classes.
Both categories suffer their own problems. The main
problem of the first category is to establish the correct re-
lationship between the person and the hard hat. Reasoning
based on the bounding box relative position seems to be too
simple to capture this, and the solutions based on distance
threshold (Chen and Demachi, 2020; Guo et al., 2020) fail
to set it properly for all cases.
The second category suffers from a major inter-class sim-
ilarity problem. Both classes, the person wearing a hard hat
and the person not wearing it, are in fact a subclass of a per-
son class. This problem is well known in subcategory classi-
fication (Cai et al., 2017; Luo et al., 2019; Han et al., 2019) as
it is harder to develop a model that can correctly distinguish
fine details between subcategories, in this case the presence
of a hard hat. It is for this reason, the best-performing mod-
els in this group actually look for the human head instead of
the whole person (Fang et al., 2018b; Wu et al., 2019a; Wang
et al., 2020; Zhou et al., 2021), making them less suitable for
direct transfer learning from a well-trained person detection
models.
Additionally, in both cases researchers tend to disregard
situations where a person is partially detected or head is ob-
structed, and it is not possible to tell if a person is wearing
head protection. In the majority, that person is incorrectly
classified as a worker without hard hat. A good example
could be Pictor-v3 dataset provided by Nath et al. (2020), in
which even a person visible from waist down is annotated as
hard hat non-wearer when in reality it cannot be determined.
This is a serious oversight and makes it impossible to apply
such a solution in practice, as the real problem is to reliably
find people who are not following the safety rules.
To address challenges of solutions based on hard hat and
person detection and incorrect partial object classification,
we are introducing a new approach to detecting hard hat wear-
ers based on head keypoint localization (Fig. 1). In the con-
text of deep learning, keypoints are understood as points of
interest in the image. Their strongest advantage is that they
are invariant for transformations, so they will not be affected
by scaling.
The most common keypoint application is human pose
estimation, where they represent human joints. However,
instead of using an existing human pose estimation model
like Chen and Demachi (2020) we define only one keypoint
representing the localization of the human head. This model
formulation enables us to correctly establish the relationship
between hard hat and hard hat wearer, with a simple rule
based algorithm presented in Algorithm 1.
3.2. Implementation
The proposed solution was implemented based on Gen-
eralized Region-based Convolutional Neural
Network (Fig. 2) framework described for the first time in
Mask R-CNN paper by He et al. (2017). This natural and
flexible extension to Faster R-CNN (Ren et al., 2017) en-
ables the creation of models capable of performing a variety
of tasks simultaneously. In this case, it is object detection
Wójcik et al.: Preprint submitted to Elsevier Page 3 of 15