# Data Sources Data sources supply the rows a page group turns into pages. A group can have several; add them on its **Data sources** page with **Add data source**, pick a **Type**, fill in the fields that type shows, and save. Rank reads the sources in order: when two sources hold the same slug, the first one wins. Every source also has a **Cache duration (seconds)**; see [Loading and refreshing the data](#loading-and-refreshing-the-data) for what it does and why a new or changed source needs a reload before any page resolves. ## The data a source must yield Every source resolves to a list of rows, each an object of named fields. A row's `slug` field is the part of the URL that replaces `{slug}` in the pattern, so every row that should become a page needs one: ```json [ { "slug": "new-york", "name": "New York", "state": "NY", "population": "8.3M" }, { "slug": "los-angeles", "name": "Los Angeles", "state": "CA", "population": "3.9M" } ] ``` Slugs should be unique: within one source only the first row with a given slug is kept. The other fields are what the [mappings](/docs/rank/page-groups/mappings.md) write into the page. Notion rows without a `slug` property take the slug from their title. Any other row without a slug stays in the data but becomes no page and no sitemap entry. ## Source types JSON URL, CSV URL and REST API fetch only addresses WordPress accepts as safe: an `http` or `https` address on port 80, 443 or 8080, or on the site's own port for the site's own host. Another host that resolves into a private or local network fails the reload unless the `http_request_host_is_external` filter allows it. ### JSON URL Fetches a JSON array from a URL. Use it for any service that publishes its data as JSON. - **URL**: the address of the JSON document ### JSON file Reads a JSON array from a file inside the active theme's directory, so the data can live in version control beside the theme. Large files are read one row at a time, and a reload of an unchanged file only renews the cached rows' freshness. - **File path**: the path relative to the theme directory, such as `data/cities.json`. A path outside the theme is refused. ### CSV URL Fetches a CSV file from a URL and turns each line into a row. - **URL**: the address of the CSV file - **Delimiter**: the column separator, a comma by default - **Use the first row as headings**: when on, the first line names the fields; when off, the fields are numbered ### CSV file Reads a CSV file inside the active theme's directory; otherwise it works like CSV URL. - **File path**: the path relative to the theme directory, such as `data/products.csv` - **Delimiter** and **Use the first row as headings**, as above ### REST API Requests an API with a chosen method and authentication, and finds the list inside the response. - **URL**: the endpoint - **Method**: `GET` or `POST` - **Authentication**: **None**, **Bearer token** (sent as `Authorization: Bearer `) or **API key** (sent in a header you name) - **Authentication token**: the token or key, required unless authentication is **None**. A saved token is never shown or returned again; leave the field blank to keep it, or type a new one to replace it. Agent Sync leaves tokens and request headers out of the page group files it writes to the theme and keeps the saved ones when it reads a file back. - **API key header**: the header that carries an API key, `X-API-Key` by default - **JSON path**: an optional dot path to the list inside the response, such as `data.items`. A path that does not exist fails the reload. ### Google Sheets Reads one tab of a Google Sheet; each row becomes a row of data, named by the header row. - **Spreadsheet ID**: the sheet's ID, the long string between `/d/` and `/edit` in its address. The full address works too. - **Sheet name**: the tab to read, `Sheet1` by default - **Use the Google Sheets API**: off, Rank downloads the tab through Google's public CSV export; on, it reads the tab through the Google Sheets API with the key from [Settings](/docs/rank/settings.md) Both ways reach only a sheet Google serves without signing in, so share it as **Anyone with the link** (viewer). An API key identifies a Google Cloud project and grants no access to private files, and Rank has no Google sign-in, so a private sheet cannot be read either way. ### Notion Queries a Notion database; each entry becomes a row, named by its properties. - **Database ID**: the database's ID, or its full address The Notion integration token from [Settings](/docs/rank/settings.md) must be stored first, and the integration must be shared with the database. Rank turns these property types into plain values; other types are left out: - **title** and **rich_text**: plain text - **number**: the number - **select**: the option's name - **multi_select**: the option names, comma-separated - **url** and **email**: the string - **checkbox**: `true` or `false` - **date**: the start date as an ISO string ## Loading and refreshing the data Rank never reads a data source while it serves a page. It stores each source's rows in its own indexed table, `{prefix}sleek_rank_items`, and the generated pages, the sitemap, the preview and the **Data** view all read that store. A source whose rows have not been stored yet resolves no pages and adds nothing to the sitemap. Nothing fills the store when you create a group or change a source. After saving, load the data with one of these: - **Reload data** on the group's **Data sources** page or its **Data** view - `wp sleek rank reload `, or `wp sleek rank reload --all` for every published group - `wp sleek rank cache sync --group=`, or `wp sleek rank cache sync` for every published group A reload fetches every source of the group in order and replaces each source's rows in one step, so a failed fetch leaves the previous rows in place. The first source that fails stops the reload with its error; the sources before it are already reloaded. The result shows as a notice, and under **Reload data** Rank says how many rows it skipped because an earlier row has the same slug. The stored rows belong to the source's exact configuration, its cache duration included. Changing any field of a source points the group at a new, empty cache, so reload after every change. Rows that no page group refers to any more are removed by Rank's twice-daily cleanup, and a minute after a group is saved or deleted. **Cache duration (seconds)**, `3600` by default, sets how long the stored rows count as fresh. Once they are older, pages keep being served from them while Rank schedules one WP-Cron refresh of that source, between 30 seconds and five and a half minutes later. The scheduled event names the source by its cache key only; the refresh reads the source's settings, its token included, from the page group when it runs. Without a working WP-Cron, stale rows are refreshed only by a reload or `wp sleek rank cache sync`. A duration of `0` keeps the rows until you reload. The **Data** view lists the stored rows of the group fifty at a time, with every field of the page's rows as a column; its search matches slugs and field values across every row, and its slug links open the row's preview. The page and the search are part of the address, so a link opens the same rows. When the rows cannot be read, the table's card says so with **Retry**. `wp sleek rank items ` prints the same rows, and `wp sleek rank cache status` reports the active caches and the expired rows waiting for cleanup (see [CLI](/docs/rank/cli.md)). ## Configuring sources from WP-CLI or REST `wp sleek rank page-group create` and `update` take the sources as a JSON array through `--data-sources`, and the `PageGroups_create` and `PageGroups_update` operations take the same array as `dataSources`. Each entry has a `type`, a `config` object and a `cacheDuration`: | Type | `config` fields | | --------------- | ------------------------------------------------------------------------------------------------------------------------ | | `json_url` | `url`, optional `headers` | | `json_file` | `path` | | `csv_url` | `url`, `delimiter`, `hasHeader` | | `csv_file` | `path`, `delimiter`, `hasHeader` | | `rest_api` | `url`, `method`, `authType` (`none`, `bearer`, `api_key`), `authToken`, `authHeaderName`, `jsonPath`, optional `headers` | | `google_sheets` | `spreadsheetId`, `sheetName`, `useApi` | | `notion` | `databaseId` | The URL-based types and Google Sheets and Notion also accept `timeout`, the seconds a request may take before it fails, 30 by default. The workspace does not show `headers` or `timeout`. A plugin can add its own source type through the `sleekRank/register/dataSources` action (see [Hooks](/docs/rank/hooks.md)). The workspace editor offers only the seven built-in types, so configure a custom type through WP-CLI or REST.