a photon to phonon code
This AcoustSee repository contains javascript algorithms tgat aims to transform a visual environment, from a camera for example, into a intuitive soundscape in a synesthesia transform, empowering the user to experience the visual world by audio cues.
Why? We believe in solving real problems with open-source software in a fast, accessible, and impactful way. You are invited to join us to improve and make a difference!
Our synesthesia ins this project is the translation from a visual to a sound, with a user experiencie for a visually challenged user. With this concept in mind the approach is to aid a user navigation and synthesia acustical visual perception in real time. Taking a camera imput into a soundscape, where a sidewalk could have a distintive spcetrum signature that is being heard by both hears, a wall at the left with its own distintive spectrum signature syntheses another sounds, a car, a hole… a light… and so on… you catch where im going? No?, lets try another example.
Imagine a person that is unable to see, sitting at a park with headphones on and paired to a mobile phone. This phone is being weared like a necklage with the camera facing a quiet swing, as the seat of the swing gets back/further the sound generator makes a sound spectra that has less harmonics content, lower volume and wen it swings closer its spectra complexity raises.
This project aims to make this imagination into a reality, with its first milestones coded entirely by xAI Grok and now to the open source community to enhace humanity perception.
The latest stable version can be run from
Previous versions can be found at
Unstable versions currently being developed can be found at
To use it, having the most up to date version of mobile web browsers is diserable, yet most mobile internet browsers from 2021 should work.
For a complete mobile browser compability list check the doc Usage there you also find instruccions to run the first prof of concept made with Python
A mobile phone/cellphone from 2021 and up, with a front facing camera and headphones.
The webapp is designed to be used with a mobile phone where its front camera (and screen) are facing the desired objetive to be transformed in to sound, wearing the mobile phone like a necklage is its first use case in mind.
Enter https://mamware.github.io/acoustsee/present (or your version of preference from Usage)
The User Interface of the webapp is split into five regions,
IMPORTANT: The processing of the camera is done privately on your device and not a single frame is sent outside your device processor. A permision to access the camera by the browser will be requested in order to do this local processing and thus generate the audio for the navigation.
Milestone 4 (Current): Developing in Progress /future
acoustsee/
├── present/ # Current Stable Modular Webapp
│ ├── index.html
│ ├── styles.css
│ ├── main.js
│ ├── state.js
│ ├── audio-processor.js
│ ├── grid-selector.js
│ ├── ui/
│ │ ├── rectangle-handlers.js # Handles settingsToggle, modeBtn, languageBtn, startStopBtn
│ │ ├── settings-handlers.js # Manages gridSelect, synthesisSelect, languageSelect, fpsSelect
│ │ ├── frame-processor.js # Processes video frames (processFrame)
│ │ └── event-dispatcher.js # Routes events to handlers
│ └── synthesis-methods/
│ ├── grids/
│ │ ├── hex-tonnetz.js
│ │ └── circle-of-fifths.js
│ └── engines/
│ ├── sine-wave.js
│ └── fm-synthesis.js
│
├── tests/ # Unit tests (TO_DO)
│ ├── ui-handlers.test.js
│ ├── trapezoid-handlers.test.js
│ ├── settings-handlers.test.js
│ └── frame-processor.test.js
├── docs/ # Documentation
│ ├── USAGE.md
│ ├── CHANGELOG.md
│ ├── CONTRIBUTING.md
│ ├── TO_DO.md
│ ├── DIAGRAMS.md
│ ├── LICENSE.md
│ └── FAQ.md
├── past/ # Historic repository, older versions.
├── future/ # Meant to be used for fast, live testing of new features and improvements
└── README.md
Resume of TO_DO:
Diagrams covering the Turnk Based Development approach.
Reflecting: