1. What you’ll build
You’ll (1) use Algolia Crawler to pull content from several domains into one Algolia application, (2) send that content into one or more indices, and (3) expose a unified search-as-you-type UI that can be embedded in WordPress, Shopify, Webflow or any other front-end.
2. Prerequisites
| Requirement | Why you need it |
| Algolia account on the “Build” plan or higher | Access to Crawler & multi-index analytics |
| Admin API key (server side) + Search-only API key (front-end) | Security best-practice |
Access to the DNS / robots.txt / sitemap of each domain | Lets the crawler reach and follow links |
| npm / yarn (for custom InstantSearch builds) | Build the shared search UI |
| Basic HTML edit rights in every CMS | Embed the widget |
3. Create a single Algolia application
Log in → New application → choose nearest region.
In Settings › Team & Access invite stakeholders if you need separate API keys per environment.
Why single app? Algolia places no restriction on mapping sites to indices inside one application, so analytics and quota stay centralised. (support.algolia.com)
4. Plan your indexing strategy
| Scenario | Recommended setup |
| Same search UX for all sites (“global” search) | One index and add an attribute site so users can filter results. |
| Per-site UX but shared backend | One crawler with multiple actions that write to different indices (one per domain). (support.algolia.com) |
| Hard isolation (different business units) | Separate Algolia applications + their own crawlers. |
5.1 Add a crawler
Dashboard → Crawler › Create.
Give the crawler a name (e.g. multi-site-search).
5.2 Define start URLs
Add one start URL per domain, e.g.
[
"https://docs.myproduct.com/",
"https://blog.myproduct.com/",
"https://help.anotherbrand.io/"
]
5.3 Create actions (one per domain or content-type)
Each action sets:
{
"indexName": "global_content", // or "blog_index"
"pathsToMatch": ["https://blog.myproduct.com/**"],
"selectors": {
"record": {
"title": "h1",
"content": "main",
"url": "url",
"site": "static:blog" // custom attribute for faceting
}
}
}
You can edit all of this in the visual Crawler editor or paste JSON directly. (algolia.com)
5.4 Respect crawl rules
Make sure robots.txt allows Algolia’s user-agent (Algolia Crawler), or whitelist its IPs on private sites.
For password-protected staging domains, set Basic auth credentials in Crawler › Advanced settings.
5.5 Schedule & monitor
Schedule tab → choose frequency (daily, hourly…).
Use Crawler logs and Algolia CLI/API for CI-based monitoring. (algolia.com)
6. Fine-tune the indices
In Indices › Configuration set Searchable attributes (title → content).
Enable Attributes for faceting: site, language, type.
Create Synonyms for branded terms (Settings › Synonyms).
Add Rules (e.g. pin the docs homepage for query “documentation”).
7. Build a shared search UI
7.1 Choose a flavour of InstantSearch
| Environment | Library |
| Vanilla JS / Webflow embed | InstantSearch.js |
| React (Next.js, Gatsby) | React InstantSearch |
| Vue / Nuxt | Vue InstantSearch |
| Shopify | The “Algolia AI Search & Discovery” app uses InstantSearch under the hood. (algolia.com) |
7.2 Example React component with federated search
import algoliasearch from 'algoliasearch/lite';
import { InstantSearch, Index, SearchBox, Hits } from 'react-instantsearch-hooks-web';
const searchClient = algoliasearch('APP_ID', 'SEARCH_ONLY_KEY');
export default function GlobalSearch() {
return (
<InstantSearch searchClient={searchClient} indexName="global_content">
<SearchBox placeholder="Search across all sites…" />
{/* If you kept separate indices, repeat <Index> blocks */}
<Index indexName="global_content">
<h2>All results</h2>
<Hits hitComponent={Hit} />
</Index>
<Index indexName="blog_index">
<h2>Blog</h2>
<Hits hitComponent={Hit} />
</Index>
</InstantSearch>
);
}
function Hit({ hit }) {
return (
<a href={hit.url} className="block p-4 border-b">
<p className="text-sm text-gray-500">{hit.site}</p>
<h3 className="font-medium">{hit.title}</h3>
<p className="line-clamp-2">{hit.content}</p>
</a>
);
}
Styling is Tailwind-ready; adjust to match each CMS theme.
8.1 WordPress
Install WP Search with Algolia. (algolia.com)
Paste your Application ID & API keys.
In the plugin settings, disable its built-in crawler (you now control indexing centrally).
Insert the React bundle (or use the plugin’s shortcode) in your theme’s header or via Gutenberg block.
8.2 Shopify
Install the “Algolia AI Search & Discovery” app → Customize › Search → choose Custom layout → paste React bundle or tweak Liquid snippets provided by the app. (algolia.com)
8.3 Webflow
Host the compiled JS bundle on a CDN.
In Page settings › Before add the <script src="...bundle.js"></script> tag.
Optionally use No-Code Supply example for filters & pagination. (nocodesupply.co)
Server keys stay server-side. Always expose only a search-only key in your JS bundles.
Turn on HTTP referrer restrictions for Search-only keys.
Add Algolia Insights to track click & conversion events for relevance tuning.
Use Replica indices if you need per-country ranking variations.
10. Maintenance checklist
| Task | Frequency |
| Review crawler logs for blocked pages | Weekly |
| Re-run crawl after major CMS releases | On demand |
| Audit API key restrictions | Quarterly |
| Update synonyms & rules from analytics | Monthly |
Back-up index settings via Algolia CLI (algolia settings get) | Monthly |
Next steps
Explore Algolia Recommend for “related content” widgets once the indices stabilise.
If some sites expose large JSON APIs, consider bypassing the crawler and pushing data via the Batch API for faster indexing.
With this workflow you end up with one search service, one UX, many sites—maintainable, analytics-friendly and blazing-fast for users.
