Skip to main content

AWS: S3

Share your source data with us via files

Written by Kevin Jabbour

This page contains the process for sharing data with Kohort via Parquet files uploaded to an AWS S3 bucket.

Prerequisites

Before you start, you'll need:

  • A source table that contains validated, cleaned and mapped data ready for export as per the CLF data specification

  • Kohort will provide the following details:

    • S3 bucket location

    • AWS access keys

    • Your org_id and dataset_id

Upload Path and Naming Strategy

The uploads must follow the following path and naming strategy:

<org_id>/<dataset_id>/clf/v3/YYYY_MM_DD_hh_mm / product_name*.parquet

Parameters:

  • <org_id> and <dataset_id>: Unique identifiers for the organisation in conjunction with a given dataset. This therefore allows for multiple datasets within a single organisation

  • clf: The Cohort-long-form specification to which the data aligns

  • v3: Major version number of the current spec

  • YYYY_MM_DD_hh_mm: The timestamp in the path is used to group the files for a given export

  • product_name*.parquet: The product_name can be set to your liking and the * denotes the automatically numbered files after they have been partitioned

Examples:

Here’s an example of two valid file names and paths which were exported at 14:30:00 on the last day of January:

  • org_1234/source_dataset/clf/v3/2022_01_31_14_30 / best_product1.parquet

  • org_1234/source_dataset/clf/v3/2022_01_31_14_30 / best_product2.parquet

Note: Timestamp is in the file path not name.

File Creation Guidelines

To best leverage the benefits of this file format some important aspects need to be considered when creating parquet files:

  1. The files should be partitioned into smaller files so that no single file is larger than 256MB

  2. Each column should have the appropriate data type. The column field names and types should be used as detailed in the cohort long form table below

  3. Choose the correct compression algorithm. We recommend using the snappy compression algorithm when generating parquet files. This algorithm is the most widely supported by different programming language implementations

A short list of common ways to generate parquet files is listed below:

Data Source/Language

Platform/Package

Relevant Guides

Google Cloud

BigQuery

Exporting to parquet ( note compression type)

Setting column data types during export

AWS

RedShift

Exporting to parquet

Snowflake

SQL

Unloading Parquet Files

Python

Pandas

Pandas to parquet

Apache

Arrow

Reading and writing parquet

Java

SpringBoot

Writing Parquet files

Segment Guidelines

  • Limit to 250 segments to manage file transfer time and costs

  • Roll up smaller segments (e.g., instead of 200 countries, use 40 plus 'Other')

For more details, see our Introduction to Segmentation article.

Update Frequency

Ideally the Kohort system would prefer daily updates; this allows the system to keep its reporting up to date and relevant (data uploads automatically trigger an event for file processing). The exact schedule can be discussed with your implementation manager.

During implementation you may of course send test files on an ad hoc basis and move to daily processing when all setup is complete and tested.

Note: There’s no need to delete old files as Kohort will automatically remove stale files.

Did this answer your question?