tego

TEgO: Teachable Egocentric Objects Dataset

Examples

Intro

This dataset includes egocentric images of 19 distinct objects taken by two people for training and testing a teachable object recognizer. Specifically, a blind and a sighted individual takes photos of the objects using a smartphone camera to train and test their own teachable object recognizer.
A detailed description of the dataset (people, objects, environment, and lighting) can be found in our CHI 2019 paper titled “Hands holding clues for object recognition in teachable machines” (see citation below).

Dataset

You can download our full dataset here.

We also provide object heatmap annotations, manually generated by ourselves, for GTEA and GTEA Gaze+. To download the original dataset associated with these object center annotations, please visit the GTEA website and download their hand masks data.

Collection Process

Annotation Process

Images are manually annotated with hand masks, object center heatmaps, and object labels.

Examples

Structure

Note that only the training set includes hand masks and object center annotation data. In each environment folder, there are several folders, each of which contains images taken under a different condition.

JSON data structure

Two json files (labels_for_training.json and labels_for_testing.json) share the same structure. Below is the structure of the JSON data.

{
  (environment):
  {
    (method):
    {
      (image filename): (label),
      ...
    },
    ...
  },
  ...
}

Further detail

Metadata

Metadata for the testing data

Example

In the B1_PS_H_train folder, for example, the images, which include the subject’s hand, were collected by the blind subject using the screen button in the portrait mode to train a teachable object recognizer. In the B1_PS_H_test_NIL, for example, the images, which include the subject’s hand, were taken when indoor lights were turned off to test a teachable object recognizer.

Citation

Please cite our corresponding paper if you find our dataset useful. Following is the BibTex of our paper:

@inproceedings{lee2019hands,
  title={Hands Holding Clues for Object Recognition in Teachable Machines},
  author={Lee, Kyungjun and Kacorri, Hernisa},
  booktitle={Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
  year={2019},
  organization={ACM}
}

License

This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.

Funding

The work was supported, in part, by grant number 90REGE0008 (Inclusive ICT Rehabilitation Engineering Research Center), from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR), U.S. Administration for Community Living, Department of Health and Human Services.

Our results on TEgO

Following is the performance of our hand-guided object recognition system on TEgO. It shows the average recall per object in two different environments (vanilla and wild). The results were generated with our hand-guided object recognizer that was trained with images cropped by our object localization model.

Average recall per object in two different environments (vanilla and wild).  The results were generated with our hand-guided object recognizer that was trained with images cropped by our object localization model.

Contact

If you have any questions, please contact Kyungjun at kjlee@cs.umd.edu.