AI TOOLS FOR PERFORMING ARTS ARCHIVES

Discover multimodal semantic analysis tools for performing arts heritage analysis and browsing. These tools allow users to explore performances and advanced data visualization features in a VR environment. 

The tools include: 

VR visualization of performances enhanced with a semantic information layer.
A content management system with a video annotation interface.
Integrated technologies for video, audio, and language analysis.

REUSING THE SOFTWARE DEVELOPED THROUGHOUT THE PROJECT

Open-source ILPSand video analysis technologies. You are free to download and run these models and tools independently, provided you take into account their specific technical requirements and use licences.
ILSP services – Voice Activity Detection (VAD), ASR with diarization (automatic subtitles and speaker identification), Translation tools, Speech emotion recognition (SER), Text emotion analysis (TER), Music processes: https://gitlab.com/ilsp-spmd-all/premiere_group/master
Visual analysis – 3D motion mining: https://gitlab.com/ilsp-spmd-all/premiere_group/visual_analysis
CMS & HLS protocol: https://gitlab.com/ilsp-spmd-all/premiere_group/cms/dev/premiere-cms
Annotation tool: https://gitlab.com/ilsp-spmd-all/premiere_group/anno-tool

Knowledge transfer and research services provided by partners involved in Premiere. Partners are available to collaborate with institutions and artists to implement multimodal video analysis, VR video visualization (contact: Aggelos Gkiokas), 3D modelling based on markerless motion capture with cameras (contact: Alain Tremeau), and implementing CMS and archival workflows for digitized performances (contact: Rogério Ribeiro).

Motion capture video recordings will be made available through the PREMIERE Dance Motion Dataset, designed for computer vision technologies training and testing.
Description: https://premiere-project.eu/creating-a-live-performance-dataset/

MADE BY
Leader – Institute of Language and Speech Processing (ILSP) – Athena Research Centre

Creators: Athena Research Centre, , The Hubert Curien laboratory (Saint-Etienne University), Medidata.Net

Theatre and dance recordings for the tools development and testing were provided by two Portuguese art institutions. Forum Dança oversaw the documentation of dance pieces, and FITEI – Festival Internacional de Teatro de Expressão Ibérica, of the theatrical plays.

Audience: For archives, content owners, researchers, art lovers

Core technologies:

  • Multimodal semantic analysis of audiovisual materials: audio analysis (subtitles/script alignment, speaker diarization, music analysis, event description), language technologies – NLP(speech-to-text offline and real-time transcription, text and audio alignment, subtitles identification), 3D scene analysis and 3D pose trajectories estimation (2D pose estimation, light color and position​ estimation, depth/3D layout of the room​), sentiment analysis (Laban Movement Analysis of body motion, emotion recognition from face expression, text/Speech based human sentiment analysis), semantic cross linking

Application:

  • Performing arts heritage analysis – Semantic analysis of audiovisual archives & VR browsing experience including interactive data visualisation.

Multimodal analysis and XR archives browsing and visualisation tool. Premiere Demo webinar

This webinar was hosted by ‪@AthenaResearchCenter‬, ‪@UniJeanMonnetUJM‬, Medidata.net. This solution delivers the insights of semantic analysis to archive users through a VR visualisation tool.

Vassilis Katsouros on language and speach technology tools

Vassilis Katsouros from the “Athena” Research Center on language and speech technology tools in the PREMIERE project.

ADDITIONAL MATERIALS

We would like to thank all the individual artists and companies that trusted us with their work, providing video recordings of their performances for the development of the AI technologies for the archives.

This collection features seminal works from contemporary performing arts: Fra Cervello e Movimento: Extra Dry by Emio Greco (1999), Perhaps she could dance first and think afterwards by Vera Mantero (1991), Projecto Continuado / Continued Project (2015) by João dos Santos Martins; Stocos (2011) by Instituto Stocos;  Guintche (2010) by Marlene Monteiro Freitas; Fecundação e Alívio Neste Chão Irredutível onde Com Gozo Me Insurjo by Joana von Mayer Trindade & Hugo Calhim Cristóvão (2021) ; The prosfygaki by ARGO Theatre, Aimilia Ypsilandi (2022); Estro/Watts by TEP, Gonçalo Amorim (2023); História da Aranha Leopoldina by Assédio Teatro (2023); O Gatilho da Felicidade by Ana Borralho & João Galante (2017); Moria by Una Hora Menos (2021); Olo by Teatro de Ferro (2014); and Ensaio Sobre a Cegueira by Teatro o Bando (2004).