> ## Documentation Index > Fetch the complete documentation index at: https://mintlify.com/MicrosoftDocs/azure-ai-docs/llms.txt > Use this file to discover all available pages before exploring further. # Azure Open Datasets Overview > Curated public datasets optimized for machine learning, including weather, census, transportation, and economic data. # Azure Open Datasets Azure Open Datasets are curated public datasets that you can add scenario-specific features to machine learning solutions for more accurate models. Open Datasets are available in the cloud on Microsoft Azure, integrated into Azure Machine Learning, and readily available to Azure Databricks. Cleaned and structured datasets ready for ML workflows Automatically refreshed from authoritative sources Native support in Azure ML and Databricks No cost to access and use datasets ## What are Azure Open Datasets? Open Datasets are curated public datasets optimized for consumption in machine learning workflows. Datasets include public-domain data for weather, census, holidays, public safety, and location that help you train machine learning models and enrich predictive solutions. **Key Benefit**: Data scientists often spend most of their time cleaning and preparing data. Azure Open Datasets are already preprocessed and optimized, saving significant time. ## Available Dataset Categories ### Transportation Real-world transportation data for demand forecasting and logistics: Yellow taxi trip records from New York City Taxi & Limousine Commission. **Data Includes:** * Pick-up and drop-off dates/times * Pick-up and drop-off locations (latitude/longitude) * Trip distances * Itemized fares * Rate types * Payment types * Driver-reported passenger counts **Use Cases:** * Demand forecasting * Route optimization * Pricing models * Urban planning analysis ```python theme={null} from azureml.opendatasets import NycTlcYellow from datetime import datetime # Load data for specific period start_date = datetime(2023, 1, 1) end_date = datetime(2023, 1, 31) yellow_taxi = NycTlcYellow(start_date, end_date) df = yellow_taxi.to_pandas_dataframe() print(f"Loaded {len(df)} taxi trips") print(df.head()) ``` Green taxi trip records serving areas not covered by yellow cabs. **Data Includes:** * Similar structure to yellow cab data * Focus on outer boroughs * Trip and fare details * Payment information **Use Cases:** * Underserved area analysis * Multi-modal transportation planning * Fare comparison studies ```python theme={null} from azureml.opendatasets import NycTlcGreen green_taxi = NycTlcGreen(start_date, end_date) df = green_taxi.to_pandas_dataframe() ``` ### Labor and Economics Economic indicators and labor statistics: Comprehensive labor force data from the Bureau of Labor Statistics. **Data Includes:** * Labor force participation rates * Employment/unemployment rates * Demographics (age, gender, race, ethnicity) * Civilian noninstitutional population * Seasonal adjustments **Use Cases:** * Economic forecasting * Workforce planning * Regional analysis * Demographic trends ```python theme={null} from azureml.opendatasets import UsLaborForce labor_force = UsLaborForce() df = labor_force.to_pandas_dataframe() # Filter for specific demographics women_data = df[df['gender'] == 'Women'] print(women_data.head()) ``` Current Employment Statistics (CES) program data. **Data Includes:** * Nonfarm employment by industry * Average hours worked * Average hourly and weekly earnings * Production worker data * Monthly and annual data **Use Cases:** * Wage analysis * Industry trend forecasting * Economic modeling * Policy research ```python theme={null} from azureml.opendatasets import UsNationalEmployment employment = UsNationalEmployment() df = employment.to_pandas_dataframe() # Analyze tech sector employment tech_sector = df[df['industry'].str.contains('Technology')] ``` ### Weather and Climate Historical and forecast weather data: Weather observations from the National Oceanic and Atmospheric Administration. **Data Includes:** * Temperature (air, dew point) * Precipitation * Wind speed and direction * Atmospheric pressure * Visibility * Station location **Use Cases:** * Demand forecasting (energy, retail) * Agricultural planning * Predictive maintenance * Supply chain optimization ```python theme={null} from azureml.opendatasets import NoaaIsdWeather from datetime import datetime # Get weather for specific location and time start = datetime(2023, 1, 1) end = datetime(2023, 12, 31) weather = NoaaIsdWeather(start, end) df = weather.to_pandas_dataframe() # Filter for a specific station station_data = df[df['stationName'] == 'NEW YORK'] ``` ### Demographics and Census Census data including population by geography and demographics. **Data Includes:** * Population counts by geography * Age distributions * Gender breakdowns * Race and ethnicity * Housing statistics **Use Cases:** * Market segmentation * Site selection * Resource allocation * Demographic forecasting ### Public Safety Public safety incidents and responses. **Data Includes:** * Incident types and locations * Response times * Temporal patterns * Geographic distribution **Use Cases:** * Resource optimization * Predictive policing (ethical considerations required) * Emergency planning * Urban safety analysis ### Holidays Holiday calendars for multiple countries. **Data Includes:** * Holiday names and dates * Country/region * Holiday types (national, religious, etc.) **Use Cases:** * Retail demand forecasting * Workforce planning * Event scheduling * Global business operations ```python theme={null} from azureml.opendatasets import PublicHolidays holidays = PublicHolidays(start_date, end_date) df = holidays.to_pandas_dataframe() # Filter for US holidays us_holidays = df[df['countryOrRegion'] == 'United States'] ``` ## How to Access Datasets ### Python SDK Access datasets directly in Python: ```python theme={null} from azureml.opendatasets import NycTlcYellow, NoaaIsdWeather from datetime import datetime import pandas as pd # Define date range start = datetime(2023, 1, 1) end = datetime(2023, 1, 31) # Load taxi data taxi = NycTlcYellow(start, end) taxi_df = taxi.to_pandas_dataframe() # Load weather data weather = NoaaIsdWeather(start, end) weather_df = weather.to_pandas_dataframe() # Merge datasets merged = pd.merge( taxi_df, weather_df, left_on='tpepPickupDateTime', right_on='datetime', how='left' ) print(f"Combined dataset: {len(merged)} rows") ``` ### Azure Machine Learning Register datasets in your workspace: ```python theme={null} from azureml.core import Workspace, Dataset from azureml.opendatasets import NycTlcYellow # Connect to workspace ws = Workspace.from_config() # Load open dataset taxi = NycTlcYellow(start_date, end_date) taxi_df = taxi.to_pandas_dataframe() # Register as tabular dataset dataset = Dataset.Tabular.register_pandas_dataframe( dataframe=taxi_df, target=(ws.get_default_datastore(), 'taxi-data'), name='nyc-taxi-january-2023', description='NYC Yellow Taxi trips for January 2023' ) print(f"Dataset registered: {dataset.name}") ``` ### Azure Databricks Use datasets in Databricks notebooks: ```python theme={null} from azureml.opendatasets import UsLaborForce # Load dataset labor = UsLaborForce() df = labor.to_spark_dataframe() # Analyze with Spark df.createOrReplaceTempView("labor_force") result = spark.sql(""" SELECT year, AVG(unemployment_rate) as avg_unemployment FROM labor_force WHERE year >= 2020 GROUP BY year ORDER BY year """) result.show() ``` ### Azure Notebooks Access without installation: ```python theme={null} # No installation required in Azure Notebooks from azureml.opendatasets import NoaaIsdWeather from datetime import datetime start = datetime(2023, 6, 1) end = datetime(2023, 6, 30) weather = NoaaIsdWeather(start, end) df = weather.to_pandas_dataframe() # Visualize import matplotlib.pyplot as plt plt.figure(figsize=(12, 6)) plt.plot(df['datetime'], df['temperature']) plt.title('Temperature Trends - June 2023') plt.xlabel('Date') plt.ylabel('Temperature (°F)') plt.show() ``` ## Integration Patterns ### Enrich ML Models Combine Open Datasets with your data: ```python theme={null} import pandas as pd from azureml.opendatasets import NoaaIsdWeather, PublicHolidays from datetime import datetime # Your business data sales_df = pd.read_csv('sales_data.csv') sales_df['date'] = pd.to_datetime(sales_df['date']) # Load weather data start = sales_df['date'].min() end = sales_df['date'].max() weather = NoaaIsdWeather(start, end) weather_df = weather.to_pandas_dataframe() # Load holidays holidays = PublicHolidays(start, end) holidays_df = holidays.to_pandas_dataframe() # Merge all data enriched_df = sales_df.merge( weather_df[['datetime', 'temperature', 'precipitation']], left_on='date', right_on='datetime', how='left' ).merge( holidays_df[['date', 'holidayName']], on='date', how='left' ) # Add features enriched_df['is_holiday'] = enriched_df['holidayName'].notna() enriched_df['temp_range'] = pd.cut( enriched_df['temperature'], bins=[0, 40, 60, 80, 100], labels=['cold', 'cool', 'warm', 'hot'] ) print("Enhanced dataset ready for ML training") ``` ### Time Series Forecasting Use historical data for predictions: ```python theme={null} from azureml.opendatasets import NycTlcYellow from sklearn.ensemble import RandomForestRegressor import pandas as pd import numpy as np # Load historical taxi data taxi = NycTlcYellow(datetime(2022, 1, 1), datetime(2023, 12, 31)) df = taxi.to_pandas_dataframe() # Feature engineering df['hour'] = df['tpepPickupDateTime'].dt.hour df['day_of_week'] = df['tpepPickupDateTime'].dt.dayofweek df['month'] = df['tpepPickupDateTime'].dt.month # Aggregate to hourly demand hourly = df.groupby( [df['tpepPickupDateTime'].dt.date, 'hour'] ).size().reset_index(name='trip_count') # Prepare features and target X = hourly[['hour', 'day_of_week', 'month']] y = hourly['trip_count'] # Train model model = RandomForestRegressor(n_estimators=100) model.fit(X, y) # Forecast next week future_dates = pd.date_range( start=datetime.now(), periods=24*7, freq='H' ) future_X = pd.DataFrame({ 'hour': future_dates.hour, 'day_of_week': future_dates.dayofweek, 'month': future_dates.month }) predictions = model.predict(future_X) print(f"Weekly forecast: {predictions.sum():.0f} trips") ``` ## Data Processing Pipeline How Open Datasets are maintained: Data is pulled from authoritative sources at regular intervals (e.g., FTP from NOAA). Raw data is parsed into structured format (CSV, Parquet). Additional features are added (e.g., ZIP codes, nearest weather stations). Data quality checks ensure consistency and completeness. Datasets are cohosted with Azure compute for fast access. ## Request or Contribute Datasets Need data not currently available? Email the team with: * Dataset name and description * Source and licensing * Size and update frequency * Use cases [Request Dataset](mailto:aod@microsoft.com?Subject=Request%20dataset) Have a public dataset to share? Provide: * Dataset details and links * Licensing information * Update schedule * Expected growth [Contribute Dataset](mailto:aod@microsoft.com?Subject=Contribute%20dataset) ## Best Practices **Tips for working with large datasets:** * Use date filtering to load only needed data * Convert to Parquet format for faster reads * Sample during development, full data in production * Use Spark for datasets > 1GB * Cache frequently used datasets ```python theme={null} # Good: Load only needed dates taxi = NycTlcYellow(start_date, end_date) # Bad: Load entire history # taxi = NycTlcYellow() # Avoid ``` **Always validate Open Dataset data:** * Check for missing values * Verify date ranges * Validate geographic coordinates * Remove outliers * Handle timezone conversions ```python theme={null} # Check data quality print(f"Missing values:\n{df.isnull().sum()}") print(f"Date range: {df['date'].min()} to {df['date'].max()}") print(f"Outliers: {df[df['temperature'] > 150].shape[0]}") ``` **All Open Datasets are public domain:** * No personally identifiable information (PII) * Aggregated and anonymized data * Review licensing for each dataset * Cite sources in publications * Respect data provider terms ## Pricing **Azure Open Datasets are completely free to access and use.** You only pay for: * Compute resources to process data * Storage if you copy datasets * Network egress (minimal) ## Resources Browse all available datasets [View Catalog](https://azure.microsoft.com/services/open-datasets/catalog/) Complete SDK reference [API Reference](https://learn.microsoft.com/python/api/azureml-opendatasets/) Learn with examples [Start Learning](/open-datasets/how-to/create-dataset) Code examples [View Samples](https://github.com/Azure/OpenDatasetsNotebooks) ## Next Steps * [Open Datasets Overview](/open-datasets/overview) * [Create a dataset from Open Datasets](/open-datasets/how-to/create-dataset) * [Explore the dataset catalog](/open-datasets/catalog) * [Public Holidays dataset](/open-datasets/datasets/public-holidays) * [COVID-19 dataset](/open-datasets/datasets/covid-19)