A crowd/audience engagement measurement leveraging Overhead Cameras with Clustering and Euclidean Metrics.
Installation · Local Development · CLI Tools · Metrics/Code flow
Enhancing Scientific Interactions in Conferences: A Novel Approach Using Overhead Camera Pose Estimation with Clustering and Euclidean Metrics
Overhead Camera Pose Estimation: Utilizing overhead camera pose estimation, the system effectively tracks and analyzes the dynamics of scientific discussions, providing unparalleled insight into interaction patterns.
Live Interaction Analysis: Cameras strategically placed over the conference venue capture live footage of researchers during pivotal moments of their discussions, offering a real-time view into the heart of the conference.
Data-Driven Conference Improvement: Beyond merely capturing interactions, this project enables organizers to use the acquired data to make informed decisions, optimize conference layouts, and enhance the overall attendee experience.
Advanced Scoring using euclidean metrics: By assessing parameters such as proximity, duration, and frequency of interactions, it calculates a comprehensive 'interaction score' that offers quantifiable insights into the effectiveness of exchanges among researchers.
- Acquire the trained models (.pt yolo files) -> email me at [email protected] for access
- Put the downloaded models in the
./engagenet/models/
folder - Make sure the names are
best.pt
andangle_best.pt
- Clone the whole repository and navigate to the
./client
folder - Make sure you have
Node.js
andnpm/yarn
installed. - Run
npm install
in the./client
root directory to install all the dependecies
- Navigate to the
./engagenet
folder - Make sure to have
python 3.9.6
installed. - Activate a virtual env if you prefer it
- Run
pip install -r requirements.txt
![image](https://private-user-images.githubusercontent.com/78511893/267108293-3796a406-edd9-47bd-9a43-cc76e11aa714.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMzMDU0OTYsIm5iZiI6MTcyMzMwNTE5NiwicGF0aCI6Ii83ODUxMTg5My8yNjcxMDgyOTMtMzc5NmE0MDYtZWRkOS00N2JkLTlhNDMtY2M3NmUxMWFhNzE0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODEwVDE1NTMxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlYWE2ZDlmNjcyMGY2Njk0YTk4MGE5Y2M5ODc5MjgzYzQ5MDk5NWQ2MDJmMDAxMmI4MmQ3NzBhYmU3Nzc3Y2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-jOKjy6zkME1R7ZV1idwbDCX9mmWXApuCO_wncowWSg)
- I recommend having 3 different terminal windows - client, socket server and python detection code
- In the first window, navigate to the
./client
directory and runnpm run dev
to start theNextJS
app - In the second window and the same directory run
npm run start:socket
, this will start the socketIO server - Now navigate to the
./engagenet
directory and runpython main.py
⚠️ : Make sure to connect your Webcam/Camera through USB: Switch up the camera source inmain.py
if your camera is not being recognized
- Now drag the python window that pops with the live feed on top of the UI box that says place here
In the end it should look like this
If you want to use just the CLI and not the UI, navigate to the ./engagenet
directory and run python cli.py
It should end up looking like this
![image](https://private-user-images.githubusercontent.com/78511893/267115801-7af4c46b-4ac2-45aa-9b9a-383e48690d90.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMzMDU0OTYsIm5iZiI6MTcyMzMwNTE5NiwicGF0aCI6Ii83ODUxMTg5My8yNjcxMTU4MDEtN2FmNGM0NmItNGFjMi00NWFhLTliOWEtMzgzZTQ4NjkwZDkwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODEwVDE1NTMxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJjYzdiOTg1YTdmZjQxYjkzYTVmNjcyOTRlNzRlM2I2ZTI4MzU4MjQyMDUyOWViYmZmYzg5ZDM1Mjk2MmI3YmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.MiTU0zu3XBHIrYrB02RflMxA43igC_EMhXYWsPDBsOM)
- Objective: Gather and annotate a dataset for training a model to assess crowd engagement and interaction health.
- Method:
- Collect a dataset of approximately 200 crowd videos from a top-down camera perspective.
- Annotate videos with ground truth labels indicating levels of engagement or interaction health.
- Utilize this dataset for both Head Angle Calculation and Head Detection.
- Proximity Branch:
- Calculate a proximity score indicating how close individuals are to each other.
- Angle/Cluster Branch:
- Determine engagement score based on head angles and spatial clustering.
- Headcount Branch:
- Count the number of heads in the image to assess crowd size.
- Overall Scoring:
- Combine scores from the above branches for a comprehensive engagement analysis.
- Head Orientation (50% of total score):
- Count total heads (N) and heads facing towards the group (F).
- Score = (F / N) * 100.
- Proximity Analysis (40% of total score):
- Calculate distances between heads using a distance matrix.
- Identify and score head clusters based on proximity.
- Normalize scores between 0 (minimal proximity) to 1 (maximum proximity).
- Headcount Branch (5% of total score):
- Simply count the number of heads in the scene.
- Representation of head orientation angles and their corresponding directions:
Angle | Direction |
---|---|
0 | Down |
45 | Down-Right |
90 | Right |
135 | Up-Right |
180 | Up |
225 | Up-Left |
270 | Left |
315 | Down-Left |
- Distribution of files across different angle classes:
Angle | Number of Files |
---|---|
0 | 2870 |
45 | 2996 |
90 | 2806 |
135 | 2986 |
180 | 2832 |
225 | 2932 |
270 | 2800 |
315 | 2978 |