Course Syllabus for RuSSIR 2013

Jyh-Shing Roger Jang

About the course

• Title and Keywords
  • Title: QBSH and AFP as Two Successful Paradigms of Music Information Retrieval
  • Keywords: Music Information Retrieval, Query by Singing/Humming (QBSH), Audio Fingerprinting (AFP)

• Course Schedule
  • 14:00-15.30, Sept 16, 2003
    • Course introduction
    • Introduction to audio signals
    • Basic audio features
  • 14:00-15.30, Sept 17, 2003
    • Pitch tracking
  • 14:00-15.30, Sept 18, 2003
    • Comparison methods for QBSH
    • QBSH demos and MATLAB prototypes
  • 14:00-15.30, Sept 19, 2003
    • AFP feature extraction and comparison
  • 14:00-15.30, Sept 20, 2003
    • AFP optimization, prototypes and demos

• Brief Description of the course
  The course will introduce two of the most successful paradigms of content-based music information retrieval: QBSH (Query by Singing/Humming) and AFP (Audio Fingerprinting). These two paradigms have been successfully used in various companies (Shazam, Soundhound, Intonow, to name a few) to create innovation and fun applications over mobile devices. Our goal is to cover each single step of the retrieval process, such that the audience will gain a fully understand of the whole procedure. These steps include feature extraction, comparison methods, speedup techniques, optimization strategies, etc. Moreover, we shall discuss the tradeoffs between efficiency and effectiveness of the retrieval, considering the large-scale databases and limited computing power. The issue of implementation over high-computing GPU will also be address. After the course, the audience should have a comprehensive notation of how to create QBSH and AFP prototypes for their own applications. If time and space permit, we will also organize a lab session (using MATLAB) to let the audience have hands-on experiences for building components and prototypes of QBSH and AFP.

• Full Description of the course
  The course will cover two of the most successful paradigms of content-based music information retrieval, including QBSH (Query by Singing/Humming) and AFP (Audio Fingerprinting). These two paradigms have been successfully used in various companies (Shazam, Soundhound, Intonow, to name a few) to create innovation and fun applications over mobile devices. Our goal is to cover each single step of the retrieval process, such that the audience will gain a fully understand of the whole procedure. These steps include feature extraction, comparison methods, speedup techniques, optimization strategies, etc.

  For QBSH, we will cover the following topics:

  1. Audio basics: This part aims to give a common ground for those who are not familiar with audio, including audio recording specs, general characteristics, and some facts about human perception of audio.
  2. Pitch tracking: Pitch is one of the most prominent perceived features of audio, which is also the most commonly used features for QBSH. For this course, we shall introduce several important pitch tracking algorithms, from the simple peak-picking methods (such as ACF, AMDF) to the more complicated dynamic-programming-based ones (such as UPDUDP).
  3. Voiced segment detection: We need to identify the voiced segment in order to perform comparison. This part can be achieved by simple static classifiers, or by HMM (hidden Markov models) as a sequence identifier.
  4. Comparison methods: After the frame-based pitch of the query input (singing or humming) is obtain, we need to compare it with the songs in the database. Depending on the computing power, we have a spectrum of methods for comparison. The simplest method is based on string comparison of pitch difference in notes, which is quick but not so accurate. Current high-accuracy methods usually involve the use of frame-based pitch with more complicated comparisons such as linear scaling and dynamic time warping, which will be explained in details.
  5. Optimization strategies and tradeoffs: Some of the optimization strategies include progressive filtering, learning to rank, and combination of multiple comparisons. Also to deal with a large song database, sometimes we need to sacrifice accuracy for a quick response, so the tradeoff between effectiveness and efficiency is a major concern during our optimization.
  6. Demos: We shall demonstrate a QBSH website (http://mirlab.org/demo/miracle) developed by our lab. This system has around 13,000 songs in the database, and it is powered by GPU for efficient computation. We shall also demonstrate several commercially available apps based on QBSH, including Soundhound and Midomi.

  For AFP, we are going to cover the following topics:

  1. Feature extraction: We shall cover several important features for AFP, including Shazam's landmark extraction based on local peaks in the spectrogram, and Phillips' fingerprint extraction based on signs of energy differences. These features are supposed to be (more or less) invariant to signal degradation under noisy environment.
  2. Hash table construction: For efficient retrieval, we need to construct hash tables with inverted file indexing. Different features will call for different structures and access mechanisms for such hash tables.
  3. Comparison methods: Time-aligned features are required for effective comparisons, and the search strategies for time-aligned features will be discussed in this step.
  4. Optimization: We shall discuss several optimization goals in AFP, including the parameters for feature extraction, and the length of queries for retrieval.
  5. Demos: We shall demonstrate a AFP website (http://mirlab.org/demo/afp) developed by our lab. This system has around 17,000 songs in the database and its size is still increasing. Moreover, it is powered by GPU and we shall also discuss the implementation of AFP over GPU and the corresponding speedup factors. Several commercially available AFP apps (such as Shazam and Intonow) will also be introduced.

  In addition to the theoretical aspects, the course will be interleaved with MATLAB code for plotting, sound playback, and animation, to demonstrate the underlying fundamental steps of QBSH and AFP. (An example page of course material and code snippets can be found at http://mirlab.org/jang/books/audioSignalProcessing/ptTimeDomainAcf.asp?title=7-2%20ACF.)

  After the course, the audience should have a comprehensive notation of how to create QBSH and AFP prototypes for their own applications. If time and space permit, we will also organize lab sessions (using MATLAB primarily) to let the audience have hands-on experiences of building working QBSH and AFP prototypes for their own applications.

• Target Audience and expected prerequisite knowledge
  • Target Audience: Upper-division undergrads, graduate students
  • Prerequisite knowledge: Calculus, Linear Algebra, Probability theory, MATLAB (for lab session)

• Relevant references to support proposal evaluation

  • Online courses

    We have been maintaining two online courses on the web, with open course material and code snippets:

    • Audio Signal Processing and Recognition: http://mirlab.org/jang/books/audioSignalProcessing
    • Data Clustering and Pattern Recognition: http://mirlab.org/jang/books/dcpr
  • MATLAB Toolboxes

    We have been maintaining several MATLAB toolboxes that are used in the previous two online courses. These toolboxes are also used for our MIREX submissions. If we have lab sessions in the Summer School, these toolboxes will be used to create working prototypes of QBSH and AFP systems.

    • Machine Learning Toolbox: http://mirlab.org/jang/matlab/toolbox/machineLearning
    • Speech & Audio Signal Processing (SAP) Toolbox: http://mirlab.org/jang/matlab/toolbox/sap

About the Lecturer

• Biography

  Jyh-Shing Roger Jang received the B.S. degree in Electrical Engineering from National Taiwan University in 1984, and the Ph.D. degree from the EECS Department at the University of California at Berkeley in 1992. He studied artificial networks and fuzzy logic with Prof. Lotfi Zadeh, the father of fuzzy logic. As of 2013, Google Scholar shows over 7600 citations for his seminal paper on ANFIS (adaptive neuro-fuzzy inference systems), published in 1993. After obtaining his Ph.D., he joined The MathWorks to coauthor the Fuzzy Logic Toolbox used with MATLAB. Since then, he has cultivated a keen interest in implementing industry-strength software for pattern recognition and computational intelligence. He was with the CS Dept. of National Tsing Hua Univ., Taiwan, from 1995 to 2012. Since August 2012, he has been with CSIE Dept. of National Taiwan Univ., Taiwan. He has published one English book on "Neuro-Fuzzy and Soft Computing" (1997, Prentice Hall), two Chinese books on MATLAB programming, and one Chinese book on JavaScript Programming. He has also been maintaining toolboxes for machine learning and speech/audio processing, and on-line tutorials on “Data Clustering and Pattern Recognition” and “Audio Signal Processing and Recognition”. Moreover, he has been putting a significant amount of efforts to bridge the gap between academic research and industrial applications, and has achieved excellent records in technology transfer and licensing from the university to the industry. More info about Prof Jang can be found at his homepage at http://mirlab.org/jang.

• Research Statement

  Prof. Jang’s research interests include machine learning and pattern recognition, with applications to music analysis/retrieval, speech recognition/assessment/synthesis, and image identification/retrieval. He is an active participant of ISMIR (International Society of Music Information Retrieval) and has achieved excellent rankings in several tasks of MIREX (Music Information Retrieval Evaluation eXchange) for years, including query by singing/humming, audio melody extraction, beat tracking, tempo estimation, audio classification, and score following. For speech part, most of his research is focused on the use of innovative machine learning techniques for computed aided pronunciation training of Mandarin, English, Japanese, and Taiwanese.

• Contact Info

  Prof. Jyh-Shing Roger Jang
  Email: jang@mirlab.org
  Mobile: +886-966-646464
  Postal address:
  Department of Computer Science and Information Engineering
  National Taiwan University
  No. 1, Sec. 4, Roosevelt Road, Taipei, 10617 Taiwan