Free Data Sources

One of the most common questions among people starting out in data analysis in general is where to get the data they need for their work. Most people use Google as their search source, but they are unaware that there are search engines built specifically for this kind of search, such as Google's Dataset Search.

Even with a tool like that making the job easier, it is always useful to know a few references for where to get data, depending on the format and type you are looking for. So here is a list of some of the sites I use most when prospecting information for my work, whether in remote sensing, geoprocessing or building machine learning and deep learning models.

An important platform is the USGS Earth Explorer, well known among those who work with geotechnologies. It offers a wide range of data, from digital terrain models to satellite imagery.

INDE (Brazil's National Spatial Data Infrastructure) was created by the federal government and is coordinated by IBGE. Its purpose is to catalog, integrate and harmonize the geospatial data held by all national institutions.

Another important platform is where we find the images from our famous Sino-Brazilian satellite CBERS, among others. The site is designed and run by the Brazilian National Institute for Space Research, Instituto Nacional de Pesquisas Espaciais (INPE).

OpenStreetMap (OSM) is a collaborative mapping project that operates worldwide and is freely available. It is a good source when it comes to geospatial street network data.

reddit is one of those discussion forum platforms like the ones from the early days of the internet. It has plenty of datasets that can be used for initial studies without much commercial or academic purpose.

UCI is a collection of databases, domain theories and data generators used by the machine learning community for the empirical analysis of machine learning algorithms.

kaggle has long been a companion for many of us. The competition platform hosts many real and synthetic datasets, which serve as a good source for study or even as data for proofs of concept.

Five Thirty Eight is an American website that focuses on opinion poll analysis, politics, economics, sports blogging and plenty of other interesting information.

Yahoo! Finance is a very interesting finance API. It can be queried using a programming language and is widely used commercially in financial analysis.

Like the previous one, Google Finance has the same functionality and purpose and can be used as another alternative for obtaining this information.

Banco Central do Brasil (the Central Bank of Brazil) is the country's main monetary authority and provides information from several national institutions, such as the Secretaria de Moeda e Crédito (SUMOC), Banco do Brasil (BB) and the National Treasury, as well as many important indices for financial analysis.

TWB (The World Bank) is an international financial institution that provides loans and grants to the governments of low- and middle-income countries to pursue capital projects, and it holds financial data for many countries.

Quandl is an API that provides a wide range of financial information on stocks, dividends and splits for 3,000 publicly traded companies in the US and around the world.

CoinMarketCap is also an API, which makes it possible to obtain cryptocurrency market information for all kinds of coins, from the oldest ones like Bitcoin and Ether to meme coins like Doge and Shiba Inu.

Binance is the largest cryptocurrency trading platform and has an API with many tools that help us work with crypto assets.

Dados.Gov is a platform created by the Brazilian Federal Government to bring together public data and information from various institutions linked to the Federal Government.

DATASUS is the transparency platform of Brazil's public health system, the Sistema Único de Saúde (SUS). It holds a wealth of information about the national health system and has been widely used during the pandemic.

The Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP), Brazil's national institute for educational studies and research, is responsible for the school census and for administering Enem, the national high school exam, which makes it extremely important for studies related to education.

The Sistema IBGE de Recuperação Automática (SIDRA) is an IBGE platform for querying data in the institute's database of statistical tables, providing indices such as the Índice Nacional de Preços ao Consumidor Amplo (IPCA, the official consumer price index), the Pesquisa Mensal de Serviços (PMS, monthly services survey), the Índice Nacional de Preços ao Consumidor (INPC, national consumer price index) and others.

IBGE (the Brazilian Institute of Geography and Statistics) also has its own platform, which sometimes makes searching for and acquiring data easier, especially for shapefiles with the boundaries of states, cities and census tracts.

The Tribunal Superior Eleitoral (Brazil's Superior Electoral Court) also has its own platform that provides election data since 1933, including candidates, number of votes per polling section and results.

The Sistema de Inteligência Estratégica da Embrapa (Embrapa's strategic intelligence system) is a platform that offers a wealth of information about the country's agricultural sector, covering crop production, livestock, animal slaughter and more.

Mendeley is a platform used by researchers from countless institutions around the world, who make their research data available there, adding up to 29 million datasets.

As a final data source, one I came across on Twitter: the Base dos Dados API, which is unifying many of the country's databases.

It is important to point out that these are only some of the data sources I use most in my work, and I know many of my colleagues do too. Beyond that, it is always worth knowing how to do scraping in case you need to mine information from other websites, or even to get in touch with whoever manages the data; sometimes an e-mail is all it takes.