> For the complete documentation index, see [llms.txt](https://aisviz.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://aisviz.gitbook.io/documentation/default-start/quick-start.md). # Quick Start ### If you are new to AIS topics, [click here](/documentation/tutorials/automatic-identification-system.md) to learn about the Automatic Identification System (AIS). *If you are starting from scratch, download the example ".db" file from our* [*AISdb Tutorial GitHub*](https://github.com/MAPS-Lab/AISdb-Tutorials) *repository so you can follow this guide with real data.* ### Python Environment and Installation To work with the AISdb Python package, please ensure you have Python version 3.10 or higher. If you plan to use SQLite, no additional installation is required, as it is included with Python by default. Those who prefer using a PostgreSQL server must install it separately and can optionally enable the TimescaleDB extension for better performance on large time-series ingests. #### User Installation The AISdb Python package can be conveniently installed using pip. It's highly recommended that a virtual Python environment be created and the package installed within it. {% tabs %} {% tab title="Linux" icon="linux" %} {% code title="install.sh" %} ```bash python -m venv .venv # create a python virtual environment source ./.venv/bin/activate # activate the virtual environment pip install aisdb # from https://pypi.org/project/aisdb/ ``` {% endcode %} {% endtab %} {% tab title="Windows" icon="windows" %} {% code title="install.ps1" %} ```bash python -m venv .venv ./AISdb/Scripts/activate pip install aisdb ``` {% endcode %} {% endtab %} {% endtabs %} {% hint style="info" %} PyPI carries the latest stable line of AISdb. The 1.8.0-alpha release that this documentation targets is published on [GitHub](https://github.com/MAPS-Lab/AISdb/releases/tag/1.8.0-alpha) and installs from source. If you need the features introduced in 1.8.0-alpha, such as weather integration, NOAA CSV ingestion, and TimescaleDB support, follow the [Compile AISdb](broken://pages/kEZqZ6cULQ3RW0d0Hl4a) guide or run `pip install git+https://github.com/MAPS-Lab/AISdb.git` with a Rust toolchain installed. The core workflow shown in this guide works the same on both. {% endhint %} You can test your installation by running the following commands:

python
>>> import aisdb
>>> aisdb.__version__  # '1.8.0a0' when built from source, or the latest PyPI release

If you are running [Jupyter](https://jupyter.org/), ensure it is installed in the same environment as AISdb:

source ./.venv/bin/activate
pip install jupyter
jupyter notebook

The Python code in the rest of this document can be run in the Python environment you created. #### Development Installation To use *nightly builds* **(not mandatory)**, you can install AISdb from source:

source .venv/bin/activate  # On Windows use `.venv\Scripts\activate`

# Cloning the Repository and installing the package
git clone https://github.com/MAPS-Lab/AISdb.git && cd AISdb

# Windows users can instead download the installer:
#   - https://forge.rust-lang.org/infra/other-installation-methods.html#rustup
#   - https://static.rust-lang.org/rustup/dist/i686-pc-windows-gnu/rustup-init.exe
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > install-rust.sh

# Installing Rust and Maturin
/bin/bash install-rust.sh -q -y
pip install --upgrade maturin[patchelf]

# Building AISdb package with Maturin
maturin develop --release --extras=test,docs

Alternatively, you can use *nightly builds* **(not mandatory)** on **Google Colab** as follows:

import os
# Clone the AISdb repository from GitHub
!git clone https://github.com/MAPS-Lab/AISdb.git
# Install Rust using the official Rustup script
!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
# Install Maturin to build the packages
!pip install --upgrade maturin[patchelf]
# Set up environment variables
os.environ["PATH"] += os.pathsep + "/root/.cargo/bin"
# Install wasm-pack for building WebAssembly packages
!curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
# Install wasm-pack as a Cargo dependency
!cargo install wasm-pack
# Setting environment variable for the virtual environment
os.environ["VIRTUAL_ENV"] = "/usr/local"
# Change directory to AISdb for building the package
%cd AISdb
# Build and install the AISdb package using Maturin
!maturin develop --release --extras=test,docs

### Database Handling AISdb supports SQLite and PostgreSQL databases. When using PostgreSQL, [TimescaleDB](https://www.timescale.com/) is an optional extension that AISdb can take advantage of for its automatic partitioning and compression of time-series data. It is not required (PostgreSQL alone works fine), but for large ingests it is worth setting up. `decode_msgs()` exposes a `timescaledb` argument (default `False`) that switches on the TimescaleDB-aware table schema when inserting data. If you want to enable it, follow these steps first. {% stepper %} {% step %} #### Install TimescaleDB (PostgreSQL Extension) ```bash $ sudo apt install -y timescaledb-postgresql-XX # XX is the PG-SQL version ``` {% endstep %} {% step %} #### Enable the Extension in PostgreSQL ```sql > CREATE EXTENSION IF NOT EXISTS timescaledb; ``` {% endstep %} {% step %} #### Verify the Installation ```sql > SELECT * FROM timescaledb_information.version; ``` {% endstep %} {% step %} #### Restart PostgreSQL ```bash $ sudo systemctl restart postgresql ``` {% endstep %} {% endstepper %} #### Connecting to a **PostgreSQL** database This uses `psycopg`, which installs automatically with `pip install aisdb`, so there's nothing extra to add for interfacing with PostgreSQL databases. PostgreSQL accepts these [keyword arguments](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). Alternatively, a connection string may be used. Information on connection strings and PostgreSQL URI format can be found [here](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).

from aisdb.database.dbconn import PostgresDBConn

# [OPTION 1]
dbconn = PostgresDBConn(
    hostaddr='127.0.0.1',  # Replace this with the Postgres address (supports IPv6)
    port=5432,  # Replace this with the Postgres running port (if not the default)
    user='USERNAME',  # Replace this with the Postgres username
    password='PASSWORD',  # Replace this with your password
    dbname='DATABASE',  # Replace this with your database name
)

# [OPTION 2]
dbconn = PostgresDBConn('postgresql://USERNAME:PASSWORD@HOST:PORT/DATABASE')

#### Attaching a SQLite database to AISdb Querying SQLite is as easy as providing the name of a ".db" file with the same entity-relationship model as the databases supported by AISdb, which are detailed in the [SQL Database](/documentation/default-start/sql-database.md) section. We prepared an example SQLite database `example_data.db` based on AIS data from a small region near Maine, United States, in January 2022 from [Marine Cadastre](https://hub.marinecadastre.gov/pages/vesseltraffic), which is available in the AISdb [Tutorial](https://github.com/MAPS-Lab/AISdb-Tutorials) GitHub repository.

from aisdb.database.dbconn import SQLiteDBConn 

dbpath='example_data.db'
dbconn = SQLiteDBConn(dbpath=dbpath)

If you want to create your database using your data, we have a [tutorial](/documentation/tutorials/using-your-ais-data.md) with examples that show you how to create an SQLite database from open-source data. ### Querying the Database Parameters for the database query can be defined using [`aisdb.database.dbqry.DBQuery`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.database.dbqry.html#module-aisdb.database.dbqry). Iterate over rows returned from the database for each vessel with [`aisdb.database.dbqry.DBQuery.gen_qry()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.database.dbqry.html#module-aisdb.database.dbqry). Convert the results into a generator yielding dictionaries with NumPy arrays describing position vectors, *e.g.*, lon, lat, and time, using [`aisdb.track_gen.TrackGen()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.track_gen.html#aisdb.track_gen.TrackGen). The following query will return vessel trajectories from a given 1-hour time window:

import aisdb
import pandas as pd
from datetime import datetime
from collections import defaultdict

dbpath = 'example_data.db'
start_time = datetime.strptime("2022-01-01 00:00:00", '%Y-%m-%d %H:%M:%S')
end_time = datetime.strptime("2022-01-01 0:59:59", '%Y-%m-%d %H:%M:%S')

def data2frame(tracks):
    # Dictionary where for key/value
    ais_data = defaultdict(lambda: pd.DataFrame(
        columns = ['time', 'lat', 'lon', 'cog', 'rocog', 'sog', 'delta_sog']))

    for track in tracks:
        mmsi = track['mmsi']
        df = pd.DataFrame({
            'time': pd.to_datetime(track['time'], unit='s'),
            'lat': track['lat'], 'lon': track['lon'],
            'cog': track['cog'], 'sog': track['sog']
        })

        # Sort by time in descending order
        df = df.sort_values(by='time', ascending=False).reset_index(drop=True)
        # Compute the time difference in seconds
        df['time_diff'] = df['time'].diff().dt.total_seconds()
        # Compute RoCOG (Rate of Change of Course Over Ground)
        delta_cog = (df['cog'].diff() + 180) % 360 - 180
        df['rocog'] = delta_cog / df['time_diff']
        # Compute Delta SOG (Rate of Change of Speed Over Ground)
        df['delta_sog'] = df['sog'].diff() / df['time_diff']
        # Fill NaN values (first row) and infinite values (division by zero cases)
        df[['rocog', 'delta_sog']] = df[['rocog', 'delta_sog']].replace([float('inf'), float('-inf')], 0).fillna(0)
        # Drop unnecessary column
        df.drop(columns = ['time_diff'], inplace=True)
        # Store in the dictionary
        ais_data[mmsi] = df
    return ais_data

with aisdb.SQLiteDBConn(dbpath=dbpath) as dbconn:
    qry = aisdb.DBQuery(
        dbconn=dbconn, start=start_time, end=end_time,
        callback=aisdb.database.sqlfcn_callbacks.in_timerange_validmmsi,
    )

    rowgen = qry.gen_qry()
    tracks = aisdb.track_gen.TrackGen(rowgen, decimate=False)
    ais_data = data2frame(tracks)  # re-use previous function

# Display DataFrames
for key in ais_data.keys():
    print(ais_data[key])

A specific region can be queried for AIS data using [`aisdb.gis.Domain`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.gis.html#aisdb.gis.Domain) or one of its sub-classes to define a collection of `shapely` polygon features. For this example, the domain contains a single bounding box polygon derived from a longitude/latitude coordinate pair and radial distance specified in meters. If multiple features are included in the domain object, the domain boundaries will encompass the convex hull of all features.

# a circle with a 100km radius around the location point
domain = aisdb.DomainFromPoints(points=[(-69.34, 41.55)], radial_distances=[100000])

with aisdb.SQLiteDBConn(dbpath=dbpath) as dbconn:
    qry = aisdb.DBQuery(
        dbconn=dbconn, start=start_time, end=end_time,
        xmin=domain.boundary['xmin'], xmax=domain.boundary['xmax'],
        ymin=domain.boundary['ymin'], ymax=domain.boundary['ymax'],
        callback=aisdb.database.sqlfcn_callbacks.in_validmmsi_bbox,
    )
    rowgen = qry.gen_qry()
    tracks = aisdb.track_gen.TrackGen(rowgen, decimate=False)
    ais_data = data2frame(tracks)  # re-use previous function

# Display DataFrames
for key in ais_data.keys():
    print(ais_data[key])

Additional query callbacks for filtering by region, timeframe, identifier, etc., can be found in [`aisdb.database.sql_query_strings`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.database.sql_query_strings.html) and [`aisdb.database.sqlfcn_callbacks`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.database.sqlfcn_callbacks.html). ### Processing #### Voyage Modelling The above generator can be input into a processing function, yielding modified results. For example, when modeling the activity of vessels on a per-voyage or per-transit basis, each voyage is defined as a continuous vector of positions where the time between observed timestamps never exceeds 24 hours.

from datetime import timedelta

# Define a maximum time interval
maxdelta = timedelta(hours=24)

with aisdb.SQLiteDBConn(dbpath=dbpath) as dbconn:
    qry = aisdb.DBQuery(
        dbconn=dbconn, start=start_time, end=end_time,
        xmin=domain.boundary['xmin'], xmax=domain.boundary['xmax'],
        ymin=domain.boundary['ymin'], ymax=domain.boundary['ymax'],
        callback=aisdb.database.sqlfcn_callbacks.in_validmmsi_bbox,
    )
    rowgen = qry.gen_qry()
    tracks = aisdb.track_gen.TrackGen(rowgen, decimate=False)

    # Split the generated tracks into segments
    track_segments = aisdb.split_timedelta(tracks, maxdelta)
    ais_data = data2frame(track_segments)  # re-use previous function
    
    # Display DataFrames
    for key in ais_data.keys():
        print(ais_data[key])

#### Data cleaning and MMSI deduplication A common problem with **AIS data is noise**, where multiple vessels might broadcast using the same identifier (sometimes simultaneously). AISdb denoises this data in three steps. First, denoising with the encoder. The [`aisdb.denoising_encoder.encode_greatcircledistance()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.denoising_encoder.html#aisdb.denoising_encoder.encode_greatcircledistance) function checks the approximate distance between each vessel's position. It separates vectors where a vessel couldn't reasonably travel using the most direct path, such as when the implied speed exceeds 50 knots. Second, distance and speed thresholds. These thresholds limit the maximum distance or time between messages that can be considered continuous. Third, scoring and segment concatenation. A score is computed for each position delta, with sequential messages nearby at shorter intervals given a higher score. This score is calculated by dividing the Haversine distance by elapsed time. Any deltas with a score not reaching the minimum threshold are considered the start of a new segment. New segments are compared to the end of existing segments with the same vessel identifier, and if the score exceeds the minimum, they are concatenated. If multiple segments meet the minimum score, the new segment is concatenated to the existing segment with the highest score. Notice that processing functions may be executed in sequence as a chain or pipeline, so after segmenting the individual voyages as shown above, results can be input into the encoder to remove noise and correct for vessels with duplicate identifiers.

distance_threshold = 20000  # the maximum allowed distance (meters) between consecutive AIS messages
speed_threshold = 50        # the maximum allowed vessel speed in consecutive AIS messages
minscore = 1e-6             # the minimum score threshold for track segment validation

with aisdb.SQLiteDBConn(dbpath=dbpath) as dbconn:
    qry = aisdb.DBQuery(
        dbconn=dbconn, start=start_time, end=end_time,
        callback=aisdb.database.sqlfcn_callbacks.in_timerange_validmmsi,
    )
    rowgen = qry.gen_qry()
    tracks = aisdb.track_gen.TrackGen(rowgen, decimate=False)
    
    # Encode the track segments to clean and validate the track data
    tracks_encoded = aisdb.encode_greatcircledistance(tracks, 
                                                      distance_threshold=distance_threshold, 
                                                      speed_threshold=speed_threshold, 
                                                      minscore=minscore)
    ais_data = data2frame(tracks_encoded)  # re-use previous function
    
    # Display DataFrames
    for key in ais_data.keys():
        print(ais_data[key])

#### Interpolating, geofencing, and filtering Building on the above processing pipeline, the resulting cleaned trajectories can be geofenced and filtered for results contained by at least one domain polygon and interpolated for uniformity.

# Define a domain with a central point and corresponding radial distances
domain = aisdb.DomainFromPoints(points=[(-69.34, 41.55),], radial_distances=[100000,])

# Filter the encoded tracks to those inside the domain polygons
tracks_filtered = aisdb.track_gen.fence_tracks(tracks_encoded, domain)

# Interpolate the filtered tracks with a specified time interval
tracks_interp = aisdb.interp_time(tracks_filtered, step=timedelta(minutes=15))

Additional processing functions can be found in the [`aisdb.track_gen`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.track_gen.html) module. #### Exporting as CSV The resulting processed voyage data can be exported in CSV format instead of being printed:

aisdb.write_csv(tracks_interp, 'ais_processed.csv')

### Integration with external metadata AISdb supports integrating external data sources such as bathymetric charts and other raster grids. #### Bathymetric charts To determine the approximate ocean depth at each vessel position, the [`aisdb.webdata.bathymetry`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.webdata.bathymetry.html) module can be used.

import os
import aisdb

# Set the data storage directory (Gebco expects it to already exist)
data_dir = './testdata/'
os.makedirs(data_dir, exist_ok=True)

# Gebco() downloads the GEBCO bathymetry grid on first use if it is
# not already present in data_dir
bathy = aisdb.webdata.bathymetry.Gebco(data_dir=data_dir)

Once the data has been downloaded, the [`Gebco()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.webdata.bathymetry.html) class may be used to append bathymetric data to tracks in the context of a [`TrackGen()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.track_gen.html#aisdb.track_gen.TrackGen) processing pipeline like the processing functions described above.

tracks = aisdb.TrackGen(qry.gen_qry(), decimate=False)
tracks_bathymetry = bathy.merge_tracks(tracks) # merge tracks with bathymetry data

Also, see [`aisdb.webdata.shore_dist.ShoreDist`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.webdata.shore_dist.html) for determining the approximate nearest distance to shore from vessel positions. #### Rasters Similarly, arbitrary coordinate-gridded raster data may be appended to vessel tracks.

tracks = aisdb.TrackGen(qry.gen_qry(), decimate=False)
raster_path = './GMT_intermediate_coast_distance_01d.tif'

# Load the raster file
raster = aisdb.webdata.load_raster.RasterFile(raster_path)

# Merge the generated tracks with the raster data
tracks = raster.merge_tracks(tracks, new_track_key="coast_distance")

### Visualization AIS data from the database may be overlaid on a map such as the one shown below using the [`aisdb.web_interface.visualize()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.web_interface.html#aisdb.web_interface.visualize) function. This function accepts a generator of track dictionaries such as those output by [`aisdb.track_gen.TrackGen()`](https://aisviz.cs.dal.ca/AISdb/api/aisdb.track_gen.html#aisdb.track_gen.TrackGen).

from datetime import datetime, timedelta
import aisdb
from aisdb import DomainFromPoints

dbpath='example_data.db'

def color_tracks(tracks):
    ''' set the color of each vessel track using a color name or RGB value '''
    for track in tracks:
        track['color'] = 'blue' or 'rgb(0,0,255)'
        yield track

# Set the start and end times for the query
start_time = datetime.strptime("2022-01-01 00:00:00", '%Y-%m-%d %H:%M:%S')
end_time = datetime.strptime("2022-01-31 00:00:00", '%Y-%m-%d %H:%M:%S')

with aisdb.SQLiteDBConn(dbpath=dbpath) as dbconn:
    qry = aisdb.DBQuery(
        dbconn=dbconn,
        start=start_time,
        end=end_time,
        callback=aisdb.database.sqlfcn_callbacks.in_timerange_validmmsi,
    )
    rowgen = qry.gen_qry()
    
    # Convert queried rows to vessel trajectories
    tracks = aisdb.track_gen.TrackGen(rowgen, decimate=False)
    
    # Visualization
    aisdb.web_interface.visualize(
        tracks,
        visualearth=False,
        open_browser=True,
    )

Visualization of vessel tracks within a defined time range

For a complete plug-and-play solution, you may clone our [Google Colab Notebook](https://colab.research.google.com/drive/1nfDUNfw7WSa5FuxRiggbK2Rvyc5qL5iH?usp=sharing). ### Where to go next This quick start covers the basics of loading, querying, and visualizing AIS data. The [Tutorials](/documentation/tutorials/database-loading.md) section goes deeper into each step, starting with [database loading](/documentation/tutorials/database-loading.md), then [data querying](/documentation/tutorials/data-querying.md), and [data cleaning](/documentation/tutorials/data-cleaning.md) for handling noisy or duplicate position reports. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://aisviz.gitbook.io/documentation/default-start/quick-start.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.