ChartFormer is a transformative AI model designed to convert raster chart images into tactile-accessible Scalable Vector Graphics (SVGs). This tool aims to make complex data visualizations accessible for the Blind and Visually Impaired (BVI) community, facilitating interaction through tactile and audio descriptions.
The ChartFormer leverages a transformer-based architecture to analyze raster images of charts and convert them into SVG files. These files are then designed to be compatible with tactile displays and other assistive technologies, enhancing data accessibility for BVI users.
The model is trained on the Chart2Tactile dataset, a synthetic collection of 10,000 chart images across four categories: line, bar, scatter, and error-bar charts. Each image is paired with time series data and a raster version to facilitate conversion. To get the dataset, follow these steps:
-
Clone this repository:
git clone https://github.com/nsothman/ChartFormer.git -
Download the Scenegraphs folder:
https://vis.csail.mit.edu/vistext/scenegraphs.zip -
Unzip and move it to the ChartFormer folder.
-
Change the configuration to save SVG files in a destination folder (default is
ChartFormer/svg_files). -
The equivalent png charts can be found here:
https://vis.csail.mit.edu/vistext/images.zip
- AI-Driven Accessibility: Introduces a large vision language model that automatically converts chart images into tactile SVGs.
- Chart2Tactile Dataset: Provides a synthetic dataset designed following accessibility standards for training AI models in chart-to-tactile conversion.
- User-Centric Evaluation: Includes a pilot user study with BVI participants to validate the effectiveness of the tactile SVGs generated by ChartFormer.
A pilot study involving BVI participants demonstrated the practical utility of the generated SVGs. Participants engaged with tactile displays to interact with the SVGs, affirming the model's effectiveness in providing accessible data visualization.
The project and all associated materials are available on GitHub, where users can also find instructions on how to utilize the ChartFormer model for their own needs.
You can clone the repository and start exploring ChartFormer through the following link: ChartFormer GitHub Repository
Please cite the following paper if you use ChartFormer in your research or applications:
Moured, O., Alzalabny, S., Osman, A., Schwarz, T., Müller, K., & Stiefelhagen, R. (2024). ChartFormer: A Large Vision Language Model for Converting Chart Images into Tactile Accessible SVGs. In ICCHP 2024.
This research was supported by the European Research Council (ERC) and the European Union’s Horizon 2020 research and innovation program. The authors are grateful for the computing resources provided by the HoreKa computing cluster at KIT.