Using Your AIS Data
Download open-source AIS data or import your own files into AISdb, from a raw CSV through database loading, querying, cleaning, and processing.
In addition to accessing data stored on the AISdb server, you can download open-source AIS data or import your own datasets for processing and analysis using AISdb. This tutorial walks through the full pipeline for working with your own AIS files, from downloading a raw CSV and loading it into a database, through querying, cleaning, and processing the resulting tracks. We provide two loading examples, Downloading and Processing Individual Files, which demonstrates working with a small data sample and a SQLite database, and Pipeline for Bulk File Downloads and Database Integration, which outlines our approach to handling multiple data file downloads and a PostgreSQL database. From there, the tutorial covers querying your database with DBQuery, cleaning the resulting tracks, and processing them into a usable form.
Data Source
U.S. vessel traffic data across user-defined geographies and periods is available at MarineCadastre. This resource offers comprehensive AIS data that can be accessed for various maritime analysis purposes. We can tailor the dataset based on research needs by selecting specific regions and timeframes.
Downloading and Processing Individual Files
In the following example, we will show how to download and process a single data file and import the data to a newly created SQLite database.
First, download the AIS data for a single day using the curl command:
curl -o ./data/AIS_2020_01_01.zip https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2020/AIS_2020_01_01.zipThen, extract the downloaded ZIP file to a specific path:
unzip ./data/AIS_2020_01_01.zip -d ./data/We will look at the columns in the downloaded CSV file.
import pandas as pd
# Read CSV file in pandas dataframe
df_ = pd.read_csv("./data/AIS_2020_01_01.csv", parse_dates=["BaseDateTime"])
print(df_.columns)Index(['MMSI', 'BaseDateTime', 'LAT', 'LON', 'SOG', 'COG', 'Heading',
'VesselName', 'IMO', 'CallSign', 'VesselType', 'Status',
'Length', 'Width', 'Draft', 'Cargo', 'TransceiverClass'],
dtype='object')The required columns for AISdb have specific names and may differ from those in the imported dataset. Therefore, let's define the exact list of columns needed.
Next, we update the column names in the existing dataframe df_ and change the time format as required. The timestamp of an AIS message is represented by BaseDateTime in the default format YYYY-MM-DDTHH:MM:SS. For AISdb, however, the time is represented in UNIX format. We now read the CSV and apply the necessary changes to the date format:
In the code, we can see that we have mapped the column names accordingly. The data type of some columns has also been changed. An nm4 file usually contains raw messages with a Message_ID separating static messages from dynamic ones; however, the MarineCadastre data does not have such an indicator of message type. Thus, adding static messages is necessary for database creation so that a table related to metadata is created.
Let's process the CSV to create an SQLite database using the aisdb package.
A SQLite database has now been created.
If you would rather load the same file into PostgreSQL, connect with PostgresDBConn instead of SQLiteDBConn and pass the same filepaths list to decode_msgs. PostgresDBConn accepts either a set of keyword arguments or a libpq connection string.
The keyword-argument form is equivalent and often easier to keep in a config file or environment variables.
Everything downstream, querying, cleaning, and processing, works identically against a PostgreSQL connection, since DBQuery accepts any SQLiteDBConn or PostgresDBConn object.
Querying Your Own Data
Once your file is decoded and sitting in a database, the loading step is done and the rest of the pipeline is the same regardless of where the data came from. Querying uses DBQuery to select rows by time range and location, then TrackGen to reshape those rows into per-vessel track dictionaries. Both classes are covered in more depth in Data Querying; here we apply them directly to the SQLite database created above.
in_timerange_validmmsi restricts the query to the given window and drops rows with a malformed MMSI, which is a sensible default when the source data has not been cleaned yet. gen_qry() yields rows sorted by MMSI and time, and TrackGen collapses those rows into one dictionary per vessel, with decimate=True applying linear curve decimation to drop redundant points along straight segments.
Cleaning the Data
Data pulled from a real-world source such as MarineCadastre or your own receiver will contain some amount of noise, duplicate MMSIs, anchored vessels, and the occasional impossible jump in position. AISdb's denoising_encoder module handles this. The Data Cleaning tutorial covers the module in detail; a minimal pipeline on the tracks queried above looks like this.
remove_pings_wrt_speed() and encode_greatcircledistance() both accept and return a track generator, so they chain in whatever order suits the noise in your data. Since these are generators, nothing is computed until the tracks are consumed downstream.
Processing and Exporting
With the tracks queried and cleaned, the last step is turning them into something usable, whether that is an interpolated trajectory, a CSV file, or a map. aisdb.interp_time() resamples a track to a fixed time step, which is useful for aligning trajectories to a common temporal grid before further analysis.
To save the processed tracks to disk, aisdb.write_csv() writes each track's column vectors as rows in a CSV file.
Or, to inspect the result visually instead, pass the tracks straight to aisdb.web_interface.visualize(), the same way earlier tutorials do.
That covers the full pipeline for your own AIS files, loading raw messages into a database with decode_msgs, querying them back out with DBQuery and TrackGen, cleaning the result with the denoising_encoder functions, and processing the cleaned tracks into an interpolated trajectory, a CSV export, or a visualization.
Pipeline for Bulk File Downloads and Database Integration
This section provides an example of downloading and processing multiple files and loading the data into an AISdb-aligned database. The pipeline is packaged as the noaa-integrator command-line tool, available in this GitHub repository. Each stage is a subcommand; run the stages you need, in order.
Install the tool
Clone the repository and sync its environment with uv:
AIS Data Download and Extraction
The download stage fetches AIS archives from MarineCadastre for the years you specify, the organize stage groups them into {year}{month} folders, and the extract stage decompresses them. All NOAA formats are handled, including daily ZIP archives (2015-2024) and Zstandard-compressed CSV (2025 onward).
Database Loading
The load stage decodes the CSV files into an AISdb database through aisdb.decode_msgs, one month batch at a time, with automatic retries for failed batches. Both SQLite and PostgreSQL targets are supported; PostgreSQL credentials come from the --dsn argument or the NOAA_PG_DSN environment variable:
Last updated