ESTRELLA

"Adaptive Screen Capture Video Analysis"

Ugesh Egala
M.S. Thesis, University of New Mexico, 2024

This thesis presents a system for analyzing student activities during class sessions to gain insights into the learning process. A dataset consisting of 14 screen recordings, with 2 videos labeled across two stages, was used for training, validation, and testing. The methodology employs adaptive sampling, initially at 10 frames per minute to identify active regions, followed by detailed analysis at 1 frame per second using OCR to detect typing activities through character changes. The results demonstrate over a 50% reduction in computational load while maintaining high accuracy in detecting student engagement. A total of 26 hours of screen capture videos were processed, with AWS Textract achieving 87% accuracy, Tesseract 80%, EasyOCR 76%, and MMOCR 72%. By analyzing activity and typing patterns, the system significantly reduces the need for manual video review, providing valuable insights into how students interact with computers, the time spent by each group, and periods of engagement, thereby enhancing our understanding of student interaction dynamics.

"3D Speaker Geometry Inference from Digital Video using 2D Projective Geometry"

Sebastian Alonso Janampa Rojas
M.S. Thesis, University of New Mexico, 2023

The thesis discusses the need for a 3D world model reconstruction from raw video frames using 2D projective geometry. We propose a computer-aided approach to reconstructing a 3D speaker geometry from classroom videos of students learning Python.

The proposed method uses a transformer model to detect line candidates. Once the users identify lines corresponding to three orthogonal directions, the method computes the three vanishing points and the camera matrix. The method identifies the student’s mouths based on face landmark detection. After the estimates of the projections of the students’ mouths on the table are verified by the users, the proposed approach reconstructs the 3D speaker geometry for students whose mouths are visible.

The performance of the method is tested on synthetic images and real-life classroom images. The results show promising results for 3D table reconstruction. Furthermore, the 3D speaker geometry can be reconstructed without any projection corrections for the cases where the speakers are visible on the left and right sides of the table.

"Speaker Diarization and Identification from Single-Channel Classroom Audio Recording Using Virtual Microphones"

Antonio Gomez
M.S. Thesis, University of New Mexico, 2022

Speaker identification in noisy audio recordings, specifically those from collaborative learning environments, can be extremely challenging. There is a need to identify individual students talking in small groups from other students talking at the same time. To solve the problem, we assume the use of a single microphone per student group without any access to previous large datasets for training.

This dissertation proposes a method of speaker identification using cross-correlation patterns associated to an array of virtual microphones, centered around the physical microphone. The virtual microphones are simulated by using approximate speaker geometry observed from a video recording. The patterns are constructed based on estimates of the room impulse responses for each virtual microphone. The correlation patterns are then used to identify the speakers. The proposed method is validated with classroom audios and shown to substantially outperform diarization services provided by Google Cloud and Amazon AWS.

"Spanish and English Phoneme Recognition by Training on Simulated Classroom Audio Recordings of Collaborative Learning Environments"

Mario Javier Esparza Perez
M.S. Thesis, University of New Mexico, 2021

Audio recordings of collaborative learning environments contain a constant presence of cross-talk and background noise. Dynamic speech recognition between Spanish and English is required in these environments. To eliminate the standard requirement of large-scale ground truth, the thesis develops a simulated dataset by transforming audio transcriptions into phonemes and using 3D speaker geometry and data augmentation to generate an acoustic simulation of Spanish and English speech. The thesis develops a low-complexity neural network for recognizing Spanish and English phonemes (available at github.com/muelitas/keywordRec). When trained on 41 English phonemes, 0.099 PER is achieved on Speech Commands. When trained on 36 Spanish phonemes and tested on real recordings of collaborative learning environments, a 0.7208 LER is achieved. Slightly better than Google’s Speech-to-text 0.7272 LER, which used anywhere from 15 to 1,635 times more parameters and trained on 300 to 27,500 hours of real data as opposed to 13 hours of simulated audios.

"Long Term Object Detection and Tracking in Collaborative Learning Environments"

Sravani Teeparthi
M.S. Thesis, University of New Mexico, 2021

This thesis contributes robust methods for computer keyboard detection, tracking, and student hand detection. For hand detection, the thesis integrates object detection with clustering and time-projections for accurate, long-term assessment of student participation. The hand detection method was integrated into a writing detection system and can also be used for later research on recognizing student gestures.

"Fast Video-based Face Recognition in Collaborative Learning Environments"

Phuong Tran
M.S. Thesis, University of New Mexico, 2021

The thesis develops a fast method for face recognition in digital videos applicable to large datasets. It introduces several approaches to address challenges associated with video face recognition and develops the AOLME dataset of 138 student faces. Compared to the baseline method, the final optimized method achieved improved accuracy and faster recognition times. Using face prototype sampling only, the proposed method achieved an accuracy of 71.8% compared to 62.3% for the baseline system while running 11.6 times faster.