{"id":3979,"date":"2022-09-21T16:40:41","date_gmt":"2022-09-21T16:40:41","guid":{"rendered":"https:\/\/billsbud.org\/?p=3979"},"modified":"2022-09-21T16:40:41","modified_gmt":"2022-09-21T16:40:41","slug":"introducing-whisper","status":"publish","type":"post","link":"https:\/\/billsbud.org\/?p=3979","title":{"rendered":"Introducing Whisper"},"content":{"rendered":"<p><!--kg-card-begin: markdown--><\/p>\n<div class=\"js-excerpt\">\n<img decoding=\"async\" src=\"https:\/\/openai.com\/content\/images\/2022\/09\/social.png\" alt=\"Introducing Whisper\"><\/p>\n<p>We&#x2019;ve trained and are open-sourcing a neural net called Whisper that approaches human level robustness and accuracy on English speech recognition.<\/p>\n<\/div>\n<section class=\"btns\">\n  <a href=\"https:\/\/cdn.openai.com\/papers\/whisper.pdf\" class=\"btn btn-ypadded pl-0.125 d-block icon-paper\">Read Paper<\/a><\/p>\n<hr class=\"my-0\">\n  <a href=\"https:\/\/github.com\/openai\/whisper\" class=\"btn btn-ypadded pl-0.125 d-block icon-code\">View Code<\/a><\/p>\n<hr class=\"my-0\">\n  <a href=\"https:\/\/github.com\/openai\/whisper\/blob\/main\/model-card.md\" class=\"btn btn-ypadded pl-0.125 d-block icon-slides\">View Model Card<\/a><br \/>\n<\/section>\n<div class=\"js-root mt-3 mb-2 d-none\">\n<div class=\"d-flex align-items-center justify-content-between mb-0.75\">\n<h3 class=\"font-base font-sans-serif font-bold mb-0\">Whisper examples:<\/h3>\n<p>    <select class=\"font-sans-serif medium-xsmall-copy js-example-select\"><\/select>\n  <\/div>\n<div class=\"demo\">\n    <button class=\"play-button js-play-button\"><\/p>\n<div class=\"layers\">\n<div class=\"layer\"><\/div>\n<div class=\"layer\"><\/div>\n<div class=\"layer\"><\/div>\n<div class=\"layer\"><\/div>\n<\/p><\/div>\n<div class=\"play-button__inner\">\n        <span class=\"icon icon-play js-play-icon\"><\/span><br \/>\n        <span class=\"icon icon-pause js-pause-icon\" style=\"display: none;\"><\/span>\n      <\/div>\n<p>    <\/button><\/p>\n<div class=\"position-relative w-100\">\n<div class=\"soundwave soundwave--unplayed js-soundwave\"><\/div>\n<div class=\"soundwave soundwave--played js-soundwave\"><\/div>\n<\/p><\/div>\n<div class=\"demo__output font-sans-serif medium-xsmall-copy\">\n<div class=\"js-output\"><\/div>\n<p>      <button class=\"small-caps demo__reveal js-reveal\">Reveal Transcript<\/button>\n    <\/div>\n<\/p><\/div>\n<p><audio type=\"audio\/wav\" class=\"js-audio\"><\/audio><\/p>\n<\/div>\n<p>Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web.   We show that the use of such a large and diverse dataset leads to improved robustness to accents, background noise and technical language. Moreover, it enables transcription in multiple languages, as well as translation from those languages into English. We are open-sourcing models and inference code to serve as a foundation for building useful applications and for further research on robust speech processing.<\/p>\n<div class=\"d-md-none\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/asr-summary-of-model-architecture-mobile.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<div class=\"d-none d-md-block wide my-1.5\">\n<div class=\"mx-xl-auto\" style=\"max-width:780px\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/asr-summary-of-model-architecture-desktop.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<\/div>\n<p>The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.<\/p>\n<div class=\"d-lg-none\">\n<div style=\"max-width:500px\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220919a\/asr-details-mobile.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<\/div>\n<div class=\"d-none d-lg-block wide my-2\">\n<div class=\"mx-xl-auto\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220919a\/asr-details-desktop.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<\/div>\n<p>Other existing approaches frequently use smaller, more closely paired audio-text training datasets<span class=\"js-rfref\" data-id=\"simply-mix\"><\/span><span class=\"js-rfref\" data-id=\"the-peoples-speech\"><\/span><span class=\"js-rfref\" data-id=\"gigaspeech\"><\/span>, or use broad but unsupervised audio pretraining<span class=\"js-rfref\" data-id=\"self-supervised-learning\"><\/span><span class=\"js-rfref\" data-id=\"unsupervised-speech-recognition\"><\/span><span class=\"js-rfref\" data-id=\"the-frontier\"><\/span>. Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper&#x2019;s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.<\/p>\n<p>About a third of Whisper&#x2019;s audio dataset is non-English, and it is alternately given the task of transcribing in the original language or translating to English. We find this approach is particularly effective at learning speech to text translation and outperforms the supervised SOTA on CoVoST2 to English translation zero-shot.<\/p>\n<div class=\"d-lg-none\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/asr-training-data-mobile.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<div class=\"d-none d-lg-block wide my-2\">\n<div class=\"mx-xl-auto\">\n<img decoding=\"async\" src=\"https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/asr-training-data-desktop.svg\" class=\"w-100\" alt=\"Introducing Whisper\">\n<\/div>\n<\/div>\n<p>We hope Whisper&#x2019;s high accuracy and ease of use will allow developers to add voice interfaces to a much wider set of applications. Check out the <a href=\"https:\/\/cdn.openai.com\/papers\/whisper.pdf\">paper<\/a>, <a href=\"https:\/\/github.com\/openai\/whisper\/blob\/main\/model-card.md\">model card<\/a>, and <a href=\"https:\/\/github.com\/openai\/whisper\">code<\/a> to learn more details and to try out Whisper.<\/p>\n<footer class=\"post-footer js-post-footer\">\n<!-- footer item --><\/p>\n<div data-order=\"-1\">\n<hr>\n<div class=\"row\" id=\"references\">\n<div class=\"col\">References<\/div>\n<div class=\"col\">\n<ol>\n<li class=\"js-ref\" data-id=\"simply-mix\">\nChan, W., Park, D., Lee, C., Zhang, Y., Le, Q., and Norouzi, M. SpeechStew: Simply mix all available speech recogni- tion data to train one large neural network. <a href=\"https:\/\/arxiv.org\/abs\/2104.02133\">arXiv preprint arXiv:2104.02133, 2021<\/a>.\n<\/li>\n<li class=\"js-ref\" data-id=\"the-peoples-speech\">\nGalvez, D., Diamos, G., Torres, J. M. C., Achorn, K., Gopi, A., Kanter, D., Lam, M., Mazumder, M., and Reddi, V. J. The people&#x2019;s speech: A large-scale diverse english speech recognition dataset for commercial usage. <a href=\"https:\/\/arxiv.org\/abs\/2111.09344\">arXiv preprint arXiv:2111.09344, 2021<\/a>.\n<\/li>\n<li class=\"js-ref\" data-id=\"gigaspeech\">\nChen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. <a href=\"https:\/\/arxiv.org\/abs\/2106.06909\">arXiv preprint arXiv:2106.06909, 2021<\/a>.\n<\/li>\n<li class=\"js-ref\" data-id=\"self-supervised-learning\">\nBaevski, A., Zhou, H., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. <a href=\"https:\/\/arxiv.org\/abs\/2006.11477\">arXiv preprint arXiv:2006.11477, 2020<\/a>.\n<\/li>\n<li class=\"js-ref\" data-id=\"unsupervised-speech-recognition\">\nBaevski, A., Hsu, W.N., Conneau, A., and Auli, M. Unsu pervised speech recognition. Advances in Neural Information Processing Systems, 34:27826&#x2013;27839, 2021.\n<\/li>\n<li class=\"js-ref\" data-id=\"the-frontier\">\nZhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. <a href=\"https:\/\/arxiv.org\/abs\/2109.13226\">arXiv preprint arXiv:2109.13226, 2021<\/a>.\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/footer>\n<style>\n  @keyframes layer1 {\n    0% {\n      transform: rotate(0deg);\n      opacity: 0.5;\n    }\n    50% {\n      transform: rotate(180deg) scale(0.98);\n      opacity: 0.44;\n    }\n    100% {\n      transform: rotate(360deg);\n      opacity: 0.5;\n    }\n  }<\/p>\n<p>  @keyframes layer2 {\n    0% {\n      transform: rotate(0deg);\n      opacity: 0.5;\n    }\n    50% {\n      transform: rotate(180deg) scale(0.95);\n      opacity: 0.46;\n    }\n    100% {\n      transform: rotate(360deg);\n      opacity: 0.5;\n    }\n  }<\/p>\n<p>  @keyframes layer3 {\n    0% {\n      transform: rotate(0deg);\n      opacity: 0.5;\n    }\n    50% {\n      transform: rotate(180deg) scale(0.97);\n      opacity: 0.41;\n    }\n    100% {\n      transform: rotate(360deg);\n      opacity: 0.5;\n    }\n  }<\/p>\n<p>  @keyframes layer4 {\n    0% {\n      transform: rotate(0deg);\n      opacity: 0.5;\n    }\n    50% {\n      transform: rotate(180deg) scale(0.96);\n      opacity: 0.46;\n    }\n    100% {\n      transform: rotate(360deg);\n      opacity: 0.5;\n    }\n  }<\/p>\n<p>  .demo {\n    display: grid;\n    grid-auto-flow: column;\n    grid-template-columns: auto 1fr;\n    align-items: center;\n    grid-gap: 0 32px;\n  }<\/p>\n<p>  .play-button {\n    position: relative;\n    width: 54px;\n    height: 54px;\n  }<\/p>\n<p>  .layers {\n    opacity: 0.4;\n    transition: 300ms all ease;\n  }<\/p>\n<p>  .play-button:hover .layers {\n    opacity: 0.7;\n  }<\/p>\n<p>  .play-button:hover {\n    opacity: 1;\n  }<\/p>\n<p>  .layer {\n    background-color: #51DA4C;\n    width: 100%;\n    height: 100%;\n    position: absolute;\n    border-radius: 50%;\n    opacity: 0.5;\n    transition: 500ms all ease;\n  }<\/p>\n<p>  .layer:nth-child(1) {\n    top: -4px;\n    left: -4px;\n    transform-origin: calc(50% + 4px) calc(50% + 4px);\n    animation: layer1 4500ms linear infinite;\n  }<\/p>\n<p>  .layer:nth-child(2) {\n    top: 4px;\n    left: 4px;\n    transform-origin: calc(50% - 4px) calc(50% - 4px);\n    animation: layer2 4500ms linear infinite;\n  }<\/p>\n<p>  .layer:nth-child(3) {\n    top: 4px;\n    left: -4px;\n    transform-origin: calc(50% + 4px) calc(50% - 4px);\n    animation: layer3 4500ms linear infinite;\n  }<\/p>\n<p>  .layer:nth-child(4) {\n    top: -4px;\n    left: 4px;\n    transform-origin: calc(50% - 4px) calc(50% + 4px);\n    animation: layer4 4500ms linear infinite;\n  }<\/p>\n<p>  .playing .layer {\n    animation-duration: 2200ms;\n  }<\/p>\n<p>  .play-button__inner {\n    position: relative;\n    background-color: #fff;\n    width: 100%;\n    height: 100%;\n    border-radius: 50%;\n    display: flex;\n    justify-content: center;\n    align-items: center;\n    font-size: 27px;\n    line-height: 0.5;\n    color: #51DA4C;\n  }<\/p>\n<p>  .soundwave {\n    height: 44px;\n    width: 100%;\n    mask-image: url('https:\/\/ift.tt\/HzjT6bP);\n    mask-size: 100% auto;\n    mask-repeat: no-repeat;\n    mask-size: 100% auto;\n    -webkit-mask-image: url('https:\/\/ift.tt\/HzjT6bP);\n    -webkit-mask-size: 100% auto;\n    -webkit-mask-repeat: no-repeat;\n    -webkit-mask-size: 100% auto;\n  }<\/p>\n<p>  .soundwave--unplayed {\n    background-color: #d6d6d6;\n  }<\/p>\n<p>  .soundwave--played {\n    background-color: #51DA4C;\n    position: absolute;\n    top: 0;\n    left: 0;\n    clip-path: polygon(0 0, 0 0, 0 100%, 0 100%);\n  }<\/p>\n<p>  .demo__output {\n    position: relative;\n    margin-top: 24px;\n    background-color: #f2f2f2;\n    border-radius: 2px;\n    padding: 10px 16px 9px;\n    grid-row: 2;\n    grid-column: 1 \/ -1;\n  }<\/p>\n<p>  .demo__reveal {\n    color: #51DA4C;\n    text-align: center;\n    width: 100%;\n    padding: 4px 0 2px;\n  }\n<\/style>\n<p><script type=\"module\">\n  const EXAMPLES = {\n    'micromachines': {\n      label: 'Speed talking',\n      src: 'https:\/\/cdn.openai.com\/whisper\/draft-20220913a\/micro-machines.wav',\n      waveform: 'https:\/\/cdn.openai.com\/whisper\/draft-20220913a\/mask-micro-machines.svg',\n      transcript: `This is the Micro Machine Man presenting the most midget miniature motorcade of Micro Machines. Each one has dramatic details, terrific trim, precision paint jobs, plus incredible Micro Machine Pocket Play Sets. There's a police station, fire station, restaurant, service station, and more. Perfect pocket portables to take any place. And there are many miniature play sets to play with, and each one comes with its own special edition Micro Machine vehicle and fun, fantastic features that miraculously move. Raise the boatlift at the airport marina. Man the gun turret at the army base. Clean your car at the car wash. Raise the toll bridge. And these play sets fit together to form a Micro Machine world. Micro Machine Pocket Play Sets, so tremendously tiny, so perfectly precise, so dazzlingly detailed, you'll want to pocket them all. Micro Machines are Micro Machine Pocket Play Sets sold separately from Galoob. The smaller they are, the better they are.`,\n    },\n    'k-pop': {\n      label: 'K-Pop',\n      src: 'https:\/\/cdn.openai.com\/whisper\/draft-20220913a\/younha.wav',\n      waveform: 'https:\/\/cdn.openai.com\/whisper\/draft-20220913a\/mask-image-black.svg',\n      transcript: `While darkness was my everything\nI ran so hard that I ran out of breath\nNever say time\u2019s up\nLike the end of the boundary\nBecause my end is not the end`,\n    },\n    'multilingual': {\n      label: 'French',\n      src: 'https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/multilingual.wav',\n      waveform: 'https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/multilingual-mask.svg',\n      transcript: `Whisper is an automatic speech recognition system based on 680,000 hours of multilingual and multitasking data collected on the Internet. We establish that the use of such a number of data is such a diversity and the reason why our system is able to understand many accents, regardless of the background noise, to understand technical vocabulary and to successfully translate from various languages into English. We distribute as a free software the source code for our models and for the inference, so that it can serve as a starting point to build useful applications and to help progress research in speech processing.`,\n    },\n    'accent': {\n      label: 'Accent',\n      src: 'https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/scottish-accent.wav',\n      waveform: 'https:\/\/cdn.openai.com\/whisper\/draft-20220920a\/mask-scottish-accent.svg',\n      transcript: `One of the most famous landmarks on the Borders, it's three hills and the myth is that Merlin, the magician, split one hill into three and left the two hills at the back of us which you can see. The weather's never good though, we stay on the Borders with the mists on the Yildens, we never get the good weather and as you can see today there's no sunshine, it's a typical Scottish Borders day.<\/p>\n<p><span class=\"font-italic xsmall-copy\">Note: Whisper transcribed \u201cEildons\u201d as \u201cYildens\u201d<\/span>`\n    }\n  }<\/p>\n<p>  const root = document.querySelector('.js-root');\n  const audio = document.querySelector('.js-audio');\n  const reveal = document.querySelector('.js-reveal');\n  const playButton = document.querySelector('.js-play-button');\n  const playIcon = document.querySelector('.js-play-icon');\n  const pauseIcon = document.querySelector('.js-pause-icon');\n  const [unplayedSoundWave, playedSoundWave] = Array.from(document.querySelectorAll('.js-soundwave'));\n  const exampleSelect = document.querySelector('.js-example-select');\n  const output = document.querySelector('.js-output');<\/p>\n<p>  let playing = false;<\/p>\n<p>  const handlePlay = () => {\n    playIcon.style.display = 'none';\n    pauseIcon.style.display = 'block';\n    root.classList.add('playing');\n  }<\/p>\n<p>  const handlePause = () => {\n    playIcon.style.display = 'block';\n    pauseIcon.style.display = 'none';\n    root.classList.remove('playing');\n  }<\/p>\n<p>  playButton.addEventListener('click', () => {\n    if (playing) {\n      audio.pause();\n      playing = false;\n      handlePause();\n    } else {\n      audio.play();\n      playing = true;\n      handlePlay();\n    }\n  });<\/p>\n<p>  audio.addEventListener('ended', () => {\n    playing = false;\n    handlePause();\n    output.innerHTML = EXAMPLES[exampleSelect.value]?.transcript;\n    reveal.style.display = 'none';\n  });<\/p>\n<p>  audio.addEventListener('timeupdate', () => {\n    const percent = audio.currentTime \/ audio.duration;\n    playedSoundWave.style.clipPath = `polygon(0 0, ${percent * 100}% 0, ${percent * 100}% 100%, 0 100%)`;\n  });<\/p>\n<p>  exampleSelect.addEventListener('change', () => {\n    \/\/ Pause the current example\n    audio.pause(); \n    playing = false;<\/p>\n<p>    reveal.style.display = 'block';\n    output.innerText = '';<\/p>\n<p>    \/\/ Reset mask\n    playedSoundWave.style.clipPath = `polygon(0 0, 0 0, 0 100%, 0 100%)`;<\/p>\n<p>    \/\/ Update the player\n    audio.src = EXAMPLES[exampleSelect.value].src;<\/p>\n<p>    handlePause();\n    updateMasks();\n  });<\/p>\n<p>  reveal.addEventListener('click', () => {\n    reveal.style.display = 'none';\n    output.innerHTML = EXAMPLES[exampleSelect.value].transcript;\n  });<\/p>\n<p>  const buildSelect = () => {\n    const fragment = document.createDocumentFragment();<\/p>\n<p>    Object.keys(EXAMPLES).forEach((key) => {\n      const option = document.createElement('option');\n      option.value = key;\n      option.innerText = EXAMPLES[key].label;\n      fragment.appendChild(option);\n    });<\/p>\n<p>    exampleSelect.appendChild(fragment);\n  }<\/p>\n<p>  const updateMasks = () => {\n    const value = exampleSelect.value;<\/p>\n<p>    [unplayedSoundWave, playedSoundWave].forEach((el) => {\n      el.style.maskImage = `url(${EXAMPLES[value].waveform})`;\n      el.style.webkitMaskImage = `url(${EXAMPLES[value].waveform})`;\n    });\n  }<\/p>\n<p>  buildSelect();\n  updateMasks();\n  audio.src = EXAMPLES[exampleSelect.value].src;<\/p>\n<p>  root.classList.remove('d-none');\n<\/script><!--kg-card-end: markdown--><\/p>\n<p>source <a href=\"https:\/\/openai.com\/blog\/whisper\/\">https:\/\/openai.com\/blog\/whisper\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We&#x2019;ve trained and are open-sourcing a neural net called Whisper that approaches human level robustness and accuracy on English speech recognition. Read Paper View Code View Model Card Whisper examples: Reveal Transcript Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. We &hellip; <a href=\"https:\/\/billsbud.org\/?p=3979\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Introducing Whisper<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-3979","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/posts\/3979","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/billsbud.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3979"}],"version-history":[{"count":1,"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/posts\/3979\/revisions"}],"predecessor-version":[{"id":3980,"href":"https:\/\/billsbud.org\/index.php?rest_route=\/wp\/v2\/posts\/3979\/revisions\/3980"}],"wp:attachment":[{"href":"https:\/\/billsbud.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3979"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/billsbud.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3979"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/billsbud.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3979"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}