Skip to main content
Guide to Connect Amazon S3:
  1. Gather your bucket name and AWS region.
  2. For private buckets: create a read-only IAM user and generate an access key pair.
  3. (Optional) Identify a path prefix to scope discovery to a subfolder.

​
What Go Fig supports

  • File formats: CSV, TSV, Parquet, JSON (auto-detected per file from the extension)
  • Compression: .gz, .zst, .bz2 (transparently decompressed)
  • Bucket types: public (no credentials) and private (IAM access key)
  • Folder structure: top-level prefixes become collections in Go Fig and files at any depth become tables. Same-named files in different subfolders stay distinct (disambiguated by their path)
  • Query engine: Go Fig queries S3 files in place via DuckDB with no copies and no re-import. Schema is discovered on connect and refreshed automatically
You don’t pick a file format when connecting. Go Fig auto-detects each file individually so a single bucket can hold mixed formats (e.g., Parquet for large data, CSV for reference tables).

​
1) Creating an IAM User (private buckets)

Skip this section if you’re connecting a public bucket. Leave the access key fields blank in step 3. For private buckets, create a dedicated IAM user with read-only access to the specific bucket:
1

Open the IAM console

In the AWS Console, open IAM → Users.
2

Create user

Click Create user.
  • User name: gofig-s3-reader
  • Access type: Programmatic access (access keys)
3

Attach read-only bucket policy

Create and attach an inline policy that grants read access only to the bucket you’re connecting:
Replace YOUR-BUCKET-NAME with your bucket. If you want to scope access to a subfolder, change /* to /your-prefix/*.
4

Generate access keys

After creating the user, go to Security credentials → Create access key → Application running outside AWS. AWS displays the Access Key ID and Secret Access Key once so copy both.
Save the Secret Access Key securely. AWS only shows it once. If you lose it, you’ll need to rotate.
Never commit access keys to source control or share them publicly. Treat them like passwords.

​
2) (Optional) Restricted-Permission Buckets

If the bucket owner doesn’t want to grant s3:ListBucket, Go Fig supports a fallback flow: after the initial connection, you’ll be prompted for an explicit list of file paths. Each path is validated via s3:GetObject only (no listing required). This is useful when:
  • You’re consuming a partner’s bucket with tightly scoped permissions
  • You only care about a specific set of files out of a very large bucket

​
3) Connect Go Fig to S3

1

Add S3 connection

On your Organization Data page, click Add data source and select Amazon S3.
The Connect a new data source modal on the Database tab with the Amazon S3 connector tile highlighted.
2

Enter connection details

Fill in your S3 connection information:
  • Bucket name: the name of your S3 bucket (e.g., redfin-public-data). Do not include s3://.
  • Region: the AWS region where the bucket lives (e.g., us-east-1, us-west-2, eu-central-1).
  • Access Key ID: your IAM user’s access key. Leave blank for public buckets.
  • Secret Access Key: your IAM user’s secret. Leave blank for public buckets.
  • Path prefix (optional): scope discovery to a subfolder (e.g., data/housing/). Leave blank to discover the entire bucket.
Match the Region to where your bucket is hosted. Using the wrong region will slow queries and may fail.
The Amazon S3 connection form with Bucket name, Region, Access Key ID, Secret Access Key, and Path prefix fields.
3

Connect

Click Connect. Go Fig tests the connection, discovers folders and files, encrypts and stores your credentials.
If the bucket restricts s3:ListBucket, Go Fig will ask you to paste an explicit allowlist of file paths on the next step (one per line, relative to the bucket root or path prefix).
4

Browse your data

Once connected, Go Fig creates a collection per top-level folder and a table for every supported file. Start exploring from the Data page, or use the data in Flows, Boards, and AI agents.

​
How files map to Go Fig

Go Fig discovers files recursively and organizes them by folder structure:
  • Top-level folders (sales, marketing) become collections
  • Files at any depth become tables, and the original filename is preserved
  • Same-named files in different subfolders stay separate (the subfolder path is shown in the table selector)
  • Files at the bucket root go into a default collection

​
Performance

Go Fig caches downloaded file data in your organization’s managed storage so repeat queries skip re-downloading from S3 and the cache refreshes when the source file changes. For Parquet files, DuckDB pushes filters and column selection down to the file so you only read what you query. S3 tables can also be joined with data from other connectors and warehouses in a single Flow.

​
Security

Go Fig stores your AWS access keys in Google Secret Manager (never in the database or logs) and connects to S3 over TLS.
Use a dedicated, read-only IAM user scoped to just the bucket (and optionally the prefix) Go Fig should read, as shown above. Never paste production root access keys. For public buckets, leave the credential fields blank to connect anonymously.