Integration between different data formats, and between data belonging to different collections, is an ongoing challenge in the MIR field. Semantic Web tools have proved to be promising resources for making different types of music information interoperable. However, the use of these technologies has so far been limited and scattered in the field. To address this, the Polifonia project 1 is developing an ontological ecosystem that can cover a wide variety of musical aspects (musical features, instruments, emotions, performances). In this paper, we present the Polifonia Ontology Network, an ecosystem that enables and fosters the transition towards Semantic MIR.

ISMIR late breaking demo

Semantic integration of MIR datasets with the polifonia ontology network

In this paper we propose the Music Note Ontology, an ontology for modelling music notes and their realisation. The ontology addresses the relation between a note represented in a symbolic representation system, and its realisation, ie a musical performance. This work therefore aims to solve the modelling and representation issues that arise when analysing the relationships between abstract symbolic features and the corresponding physical features of an audio signal. The ontology is composed of three different Ontology Design Patterns (ODP), which model the structure of the score (Score Part Pattern), the note in the symbolic notation (Music Note Pattern) and its realisation (Musical Object Pattern).

Proceedings of the 12th workshop on ontology design and patterns (WOP 2021), online, october 24, 2021.

The music note ontology

The use of Semantic Technologies - in particular the Semantic Web - has revealed to be a great tool for describing the cultural heritage domain and artistic practices. However, the panorama of ontologies for musicological applications seems to be limited and restricted to specific applications. In this research, we propose HaMSE, an ontology capable of describing musical features that can assist musicological research. More specifically, HaMSE proposes to address sues that have been affecting musicological research for decades: the representation of music and the relationship between quantitative and qualitative data. To do this, HaMSE allows the alignment between different music representation systems and describes a set of musicological features that can allow the music analysis at different granularity levels.

Proceedings of the 1st workshop on multisensory data and knowledge (MDK 2021), Zaragoza, Spain.

The HaMSE ontology: Using semantic technologies to support music representation interoperability and musicological analysis

The HaMSE ontology

This deliverable describes the results of the WP2 Musical Heritage Knowledge Graphs in its first 12 months with respect to the task T2. 1: Ontology-based knowledge graphs for music objects. The project, and specifically WP2 and T2. 1, uses knowledge engineering as a process and methodology to build ontologies and knowledge graphs that facilitate large-scale integration of various sources of musical heritage data. In this deliverable, we focus on music objects broadly understood as artefacts carrying musical content and meaning (recordings, music notation, musical theory, musical features) rather than contextual descriptions (eg composer, year of performance). New conceptualisations, in the form of ontologies, are needed to formalise the semantics of this domain and allow the creation of interoperable knowledge graphs from datasets. To achieve this, we apply a well-known ontology design methodology, eXtreme Design (XD), which emphasises:(i) ontology reuse and modularity through the design and use of Ontology Design Patterns (ODPs); and (ii) gathering a comprehensive inventory of ontological requirements through competency questions (CQs). To implement (i), we survey existing ontologies and ODP that are relevant to the domain, and existing datasets from Polifonia’s WP1 Pilots–building on the dataset survey work from WP1 [1]–and from the Music Information Retrieval (MIR) community; to implement (ii) we follow a thorough process of collecting and normalising CQs from the stories and personas that stem from the Pilots of WP1.

D2.1: Ontology-based knowledge graphs for music objects (V1. 0)

D2.1

Structure perception is a fundamental aspect of music cognition in humans. Historically, the hierarchical organization of music into structures served as a narrative device for conveying meaning, creating expectancy, and evoking emotions in the listener. Thereby, musical structures play an essential role in music composition, as they shape the musical discourse through which the composer organises his ideas. In this paper, we present a novel music segmentation method, pitchclass2vec, based on symbolic chord annotations, which are embedded into continuous vector representations using both natural language processing techniques and custom-made encodings. Our algorithm is based on long-short term memory (LSTM) neural network and outperforms the state-of-the-art techniques based on symbolic chord annotations in the field.

Proceedings of the 1st workshop on artificial intelligence and creativity

Pitchclass2vec: Symbolic music structure segmentation with chord embeddings

Pitchclass2vec

The annotation of music content is a complex process to represent due to its inherent multifaceted, subjectivity, and interdisciplinary nature. Numerous systems and conventions for annotating music have been developed as independent standards over the past decades. Little has been done to make them interoperable, which jeopardises cross-corpora studies as it requires users to familiarise with a multitude of conventions. Most of these systems lack the semantic expressiveness needed to represent the complexity of the musical language and cannot model multi-modal annotations originating from audio and symbolic sources. In this article, we introduce the Music Annotation Pattern, an Ontology Design Pattern (ODP) to homogenise different annotation systems and to represent several types of musical objects (eg chords, patterns, structures). This ODP preserves the semantics of the object’s content at different levels and temporal granularity. Moreover, our ODP accounts for multi-modality upfront, to describe annotations derived from different sources, and it is the first to enable the integration of music datasets at a large scale.

Proceedings of the 13th workshop on ontology design and patterns - WOP 2023

The music annotation pattern

Various disconnected chord datasets are currently available for music analysis and information retrieval, but these are limited by either their size, non-openness, lack of timed information, and interoperability. Together with the lack of overlapping repertoire coverage, this limits cross-corpus studies on harmony over time and across genres, while hampering research in computational music analysis (chord recognition, pattern mining, computational creativity), where machine learning methods are notoriously eager for data. To address this gap, we propose the Chord Corpus (ChoCo), a large-scale dataset that semantically integrates harmonic data from 18 different sources in various representations and formats (Harte, Leadsheet, Roman numerals, ABC, etc.). We leverage JAMS (JSON Annotated Music Specification), a popular data structure for annotations in Music Information Retrieval, to effectively represent and enrich a variety of chord-related information (chord, key, mode, etc.) in a uniform way. On top of it, we propose a novel ontology modelling music annotations and involved entities (artists, scores, etc.), build a 30M triple knowledge graph, and find 4K+ links to other datasets (MIDI-LD, LED).

Scientific Data, Nature

ChoCo: A chord corpus and a data transformation workflow for musical harmony knowledge graphs

ChoCo

Computationally creative systems for music have recently achieved impressive results, fuelled by progress in generative machine learning. However, black-box approaches have raised fundamental concerns for ethics, accountability, explainability, and musical plausibility. To enable trustworthy machine creativity, we introduce the Harmonic Memory, a Knowledge Graph (KG) of harmonic patterns extracted from a large and heterogeneous musical corpus. By leveraging a cognitive model of tonal harmony, chord progressions are segmented into meaningful structures, and patterns emerge from their comparison via harmonic similarity. Akin to a music memory, the KG holds temporal connections between consecutive patterns, as well as salient similarity relationships. After demonstrating the validity of our choices, we provide examples of how this design enables novel pathways for combinational creativity. The memory provides a fully accountable and explainable framework to inspire and support creative professionals -- allowing for the discovery of progressions consistent with given criteria, the recomposition of harmonic sections, but also the co-creation of new progressions.

The Web Conference 2023

The Harmonic Memory: a Knowledge Graph of harmonic patterns as a trustworthy framework for computational creativity

Harmory

This paper introduces a pattern to formalise context-free grammars in OWL and how to use it for sequence classification. The proposed approach and existing methods are compared and discussed based on their complexity as well as their pragmatic applicability with examples in the music domain. 

European Semantic Web Conference 2023

Classifying sequences by combining context-free grammars and OWL ontologies

Grammar

Music similarity is an essential aspect of music retrieval, recommendation systems, and music analysis. Moreover, similarity is of vital interest for music experts, as it allows studying analogies and influences among composers and historical periods. Current approaches to musical similarity rely mainly on symbolic content, which can be expensive to produce and is not always readily available. Conversely, approaches using audio signals typically fail to provide any insight about the reasons behind the observed similarity. This research addresses the limitations of current approaches by focusing on the study of musical similarity using both symbolic and audio content. The aim of this research is to develop a fully explainable and interpretable system that can provide end-users with more control and understanding of music similarity and classification systems.

Knowledge-Based Multimodal Music Similarity

The semantic description of music metadata is a key requirement for the creation of music datasets that can be aligned, integrated, and accessed for information retrieval and knowledge discovery. It is nonetheless an open challenge due to the complexity of musical concepts arising from different genres, styles, and periods – standing to benefit from a lingua franca to accommodate various stakeholders (musicologists, librarians, data engineers, etc.). To initiate this transition, we introduce the Music Meta ontology, a rich and flexible semantic model to describe music metadata related to artists, compositions, performances, recordings, and links. We follow eXtreme Design methodologies and best practices for data engineering, to reflect the perspectives and the requirements of various stakeholders into the design of the model, while leveraging ontology design patterns and accounting for provenance at different levels (claims, links). After presenting the main features of Music Meta, we provide a first evaluation of the model, alignments to other schema (Music Ontology, DOREMUS, Wikidata), and support for data transformation.

International Society for Music Information Retrieval Conference - ISMIR 2023

The Music Meta Ontology: A Flexible Semantic Model for the Interoperability of Music Metadata

Music Meta Ontology

In the music domain, several ontologies have been proposed to annotate musical data, in both symbolic and audio form, and generate semantically rich Music Knowledge Graphs. However, current models lack interoperability and are insufficient for representing music history and the cultural heritage context in which it was generated; risking the propagation of recency and cultural biases to downstream applications. In this article, we propose the Polifonia Ontology Network (PON) for music cultural heritage, centred around four modules: Music Meta (metadata), Representation (content), Source (provenance) and Instrument (cultural objects). We design PON with a strong accent on cultural stakeholder requirements and competency questions (CQs), contributing an NLP-based toolkit to support knowledge engineers in generating, validating, and analysing them; and a novel, high-quality CQ dataset produced as a result. We show current and future use of these resources by internal project pilots, early adopters in the music industry, and opportunities for the Semantic Web and Music Information Retrieval communities.

International Semantic Web Conference - ISWC 2023

The Polifonia Ontology Network: Building a Semantic Backbone for Musical Heritage

Polifonia Ontology Network

In the Western music tradition, chords are the main constituent components of harmony, a fundamental dimension of music. Despite its relevance for several Music Information Retrieval (MIR) tasks, chord-annotated audio datasets are limited and need more diversity. One way to improve those resources is by leveraging the large number of chord annotations available online, but it requires aligning them with music audio. However, existing audio-to-score alignment techniques, which typically rely on Dynamic Time Warping (DTW), fail to address this challenge, as they require weak aligned data for precise synchronisation. In this paper, we introduce ChordSync, a novel conformer-based model designed to seamlessly align chord annotations with audio, eliminating the need for weak alignment. We also provide a pretrained model and a user-friendly library, enabling users to synchronise chord annotations with audio tracks effortlessly. In this way, ChordSync creates opportunities for harnessing crowd-sourced chord data for MIR, especially in audio chord estimation, thereby facilitating the generation of novel datasets. Additionally, our system extends its utility to music education, enhancing music learning experiences by providing accurately aligned annotations, thus enabling learners to engage in synchronised musical practices.

Sound and Music Computing Conference - SMC 2024

ChordSync: Conformer-Based Alignment of Chord Annotations to Music Audio

ChordSync

We address the task of ontology learning by combining the structured NeOn methodology framework with Large Language Models (LLMs) for translating natural language domain descriptions into Turtle syntax ontologies. The main contribution of the paper is a prompt pipeline tailored for domain-agnostic modeling, exemplified through the application to a domain-specific case study: the wine ontology. The resulting pipeline is used to develop NeOn-GPT, a workflow for automatic ontology modeling, and its proof of concept implementation, integrated on top of the metaphactory platform. NeOn-GPT leverages the systematic approach of the NeOn methodology and LLMs’ generative capabilities to facilitate a more efficient ontology development process. We evaluate the proposed approach by conducting comprehensive evaluations using the Stanford wine ontology as the gold standard. The obtained results show, that LLMs are not fully equipped to perform procedural tasks required for ontology development, and lack the reasoning skills and domain expertise needed. Overall, LLMs require integration with the workflow or trajectory tools for continuous knowledge engineering tasks. Nevertheless, LLMs can significantly alleviate the time and expertise needed. Our code base is publicly available for research and development purposes, accessible at: https://github.com/andreamust/NEON-GPT.

International Semantic Web Conference - ISWC 2024

NeOn-GPT: A Large Language Model-Powered Pipeline for Ontology Learning

NeOn-GPT

The digital humanities is a diverse area of research, encompassing just as diverse data as disciplines, thus posing particular challenges for ontology engineering. This paper pushes the application of ontology design patterns to solve these recurring challenges. It proposes a set of potential generalised patterns identified in various DH-related case studies, which address recurring modelling issues in the DH domain: annotations, knowledge provenance, or the interpretation of cultural artefacts. Functioning as a proofof-concept, the work showcases how patterns could improve interoperability and knowledge capturing in ontologies for the DH.

Pubblications