TED Talk Engagement Tracking | Secure Research Platform

Executive Summary

TED Talk engagement tracking with the TED-it Experiment Website—secure, scalable, real-time analytics, AI insights, and data privacy for academic research.

The TED-it Experiment Website facilitated an academic study on user engagement with TED Talks. This project provided a secure, scalable, and research-focused platform where participants watched videos while the system logged their interactions for analysis. The platform ensured controlled access, detailed data tracking, and real-time monitoring to support structured experimentation.

A single website with 12 dynamically controlled URLs accommodated different experimental conditions. The platform featured secure authentication via Qualtrics survey integration, real-time interaction logging using Kafka and Flink, and an advanced admin dashboard leveraging Elasticsearch for data analysis. The system securely handled user data while offering researchers actionable insights into engagement trends.

The Client and Their Challenges

Researchers initiated this project to support an academic study on TED Talk engagement. They needed a web-based platform that tracked user interactions with video content while maintaining a controlled research environment. Several challenges required solutions to meet the study’s objectives. The platform had to ensure that only authorised participants accessed it through a Qualtrics survey link while preventing unauthorised usage. It needed to capture and store detailed user interactions, including play/pause actions, scrolling behaviour, and engagement duration, without compromising performance. The system had to dynamically assign participants to 12 different experimental conditions based on URL parameters. Efficient data storage and processing played a critical role in managing large volumes of user interaction data while ensuring fast query performance for research analysis. Researchers also needed an intuitive dashboard to track user activity, generate reports, and analyse engagement trends in real time. Compliance with data privacy regulations remained essential, requiring the secure storage and anonymisation of user data.

Project Details

The team developed a web-based application using Angular for the frontend and Python Django for the backend. The development period spanned from February 2021 to April 2021, and the budget accommodated SME-friendly pricing with scalability for future growth. The research team selected our services due to our expertise in developing controlled-access research platforms tailored for academic studies. Our strong background in data management, analytics, and secure web applications allowed for efficient tracking and storage of user interactions. Additionally, our ability to build an intuitive and real-time monitoring dashboard enabled researchers to analyse engagement trends seamlessly. Our commitment to data privacy and security compliance reinforced their trust in our solution.

The Solutions

The team built a single, parameterised website with 12 dynamically controlled URLs. JWT authentication restricted access to verified participants from Qualtrics. A modular, responsive frontend created a seamless user experience using Angular. For clickstream data logging, the system integrated Apache Kafka for real-time event streaming of user interactions. Apache Flink processed clickstream data before storing it in PostgreSQL, capturing detailed engagement metrics such as play rate, pause frequency, and watch duration.

The team extracted TED Talk metadata from Kaggle datasets and optimised a PostgreSQL schema for efficient video detail queries. REST API endpoints supported future data expansions and integrations. The Angular-based admin dashboard provided real-time data visualisation, with Elasticsearch enabling instant data querying and analysis. Export features allowed researchers to download engagement data in CSV and JSON formats for further analysis.

Technology & Stack Benefits

The platform’s frontend used Angular, ensuring a modular and high-performance user interface. Python Django powered the backend, delivering secure, scalable, and rapid API development. PostgreSQL optimised data handling with indexing and partitioning. AWS hosting with auto-scaling EC2 instances ensured high availability and scalability.

Key Features Implemented

The platform included a single, dynamic website supporting 12 experimental conditions. JWT authentication, integrated with Qualtrics, ensured secure participant access. Kafka and Flink enabled real-time clickstream data logging, while PostgreSQL optimised high-performance querying. The Elasticsearch-backed admin dashboard provided real-time research insights. Engagement data was exportable in CSV and JSON formats for detailed analysis.

The Result

The single-website, multi-URL approach streamlined deployment while maintaining research integrity. High-resolution interaction tracking provided researchers with valuable behavioural insights. The admin dashboard allowed researchers to monitor user activity and dynamically adjust experiment parameters. Security measures, including JWT authentication and anonymised data storage, ensured compliance with privacy regulations.

Lessons Learned

Using real-time data streaming with Kafka significantly improved the accuracy and granularity of user engagement insights. Implementing Elasticsearch enabled near-instant querying of large datasets, enhancing research efficiency. Ensuring smooth Qualtrics integration with JWT authentication proved essential for maintaining controlled access. A single parameterised site for multi-condition experimentation reduced complexity and deployment overhead.

Next Steps

Future plans include expanding the TED Talk dataset by integrating additional metadata sources. AI-driven insights will enhance the understanding of user engagement patterns. WebSockets will improve real-time tracking for more precise event monitoring. Automated reporting features will help researchers generate structured insights.

Conclusion

The TED-it Experiment Website successfully met the research requirements, enabling a controlled study on user engagement with TED Talks. Through real-time data tracking, parameterised URL-based experimentation, and an advanced admin dashboard, the platform provided researchers with valuable insights while ensuring security and compliance. This case study demonstrates how modern web technologies enhance academic research through structured data collection, scalable experimentation, and efficient data analysis.

Are you looking for a secure, scalable, and data-driven research platform? Our expertise in web-based academic research solutions ensures high-performance data tracking, real-time analytics, and secure access control. Get in touch with us today to discuss how we can support your research needs and take your study to the next level.