Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 1 addition & 29 deletions doc/user/content/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,42 +12,14 @@ aliases:

## Overview

Sources describe external systems you want Materialize to read data from, and
provide details about how to decode and interpret that data. A simplistic way to
think of this is that sources represent streams and their schemas; this isn't
entirely accurate, but provides an illustrative mental model.

In terms of SQL, sources are similar to a combination of tables and
clients.

- Like tables, sources are structured components that users can read from.
- Like clients, sources are responsible for reading data. External
sources provide all of the underlying data to process.

By looking at what comprises a source, we can develop a sense for how this
combination works.

[//]: # "TODO(morsapaes) Add details about source persistence."

## Source components

Sources consist of the following components:

Component | Use | Example
---------------|---------------------------------------------------------------------------------------------------|---------
**Connector** | Provides actual bytes of data to Materialize | Kafka
**Format** | Structures of the external source's bytes, i.e. its schema | Avro
**Envelope** | Expresses how Materialize should handle the incoming data + any additional formatting information | Upsert
{{% include-headless "/headless/source-definition" %}}

### Connectors

Materialize bundles native connectors for the following external systems:

{{< include-md file="shared-content/multilink-box-native-connectors.md" >}}

For details on the syntax, supported formats and features of each connector, check out the dedicated `CREATE SOURCE` documentation pages.


## Sources and clusters

Sources require compute resources in Materialize, and so need to be associated
Expand Down
14 changes: 14 additions & 0 deletions doc/user/content/headless/source-definition.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
headless: true
---
A source in Materialize represents an external data source. More concretely, it
specifies the connection and the ingestion configuration to use for a particular
external data source (e.g., PostgreSQL, Kafka). For those familiar with
PostgreSQL's foreign servers and foreign tables, a source is like a foreign
server, and the tables (or subsources) created from the source are like foreign
tables.

Before creating a source in Materialize, you must ensure that the external data
source is properly configured and accessible so that Materialize can establish a
connection and ingest its data. The exact configuration depends on the type of
data source.
2 changes: 1 addition & 1 deletion doc/user/content/sql/create-source/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ menu:
identifier: 'create-source'
---

A [source](/concepts/sources/) describes an external system you want Materialize to read data from, and provides details about how to decode and interpret that data.
{{% include-headless "/headless/source-definition" %}}

## Syntax summary

Expand Down
Loading