@plantagoai/seeders

Schema-driven data seeding framework. Load data from JSON, CSV, or REST APIs, validate against Zod schemas, and batch-write to Firestore. Includes a curated registry of free, legally-safe data sources.

Consumers: HerbPulse, Nomadex, Foundation

Installation

"@plantagoai/seeders": "file:../../shared/packages/seeders"

Peer Dependencies

Defining Seeds

import { defineSeed } from "@plantagoai/seeders";
import { z } from "zod";

const herbSchema = z.object({
  name: z.string().min(1),
  latin_name: z.string(),
  category: z.enum(["herb", "essential_oil", "supplement"]),
  properties: z.array(z.string()),
  safety_notes: z.string().optional(),
});

const herbSeed = defineSeed({
  name: "herbs",
  collection: "herbs",
  tag: "herb-seed-v1",       // tag for cleanup
  schema: herbSchema,
  source: {
    type: "api",
    url: "https://api.example.com/herbs",
    headers: { "Authorization": "Bearer ..." },
    rateLimit: 100,          // ms between requests
    maxRetries: 3,
    paginate: { nextKey: "next_page", dataKey: "results" },
  },
  transform: (raw) => ({
    name: raw.common_name,
    latin_name: raw.scientific_name,
    category: "herb",
    properties: raw.uses || [],
    safety_notes: raw.warnings,
  }),
});

SeedConfig

Data Sources

JSON

// From local file
const seed = defineSeed({
  ...config,
  source: { type: "json", path: "./data/herbs.json", rootKey: "items" },
});

// From URL
const seed = defineSeed({
  ...config,
  source: { type: "json", url: "https://example.com/data.json" },
});

CSV

const seed = defineSeed({
  ...config,
  source: {
    type: "csv",
    path: "./data/cities.csv",
    delimiter: ",",     // default: ","
    headers: true,      // first row is headers (default: true)
  },
});

REST API

const seed = defineSeed({
  ...config,
  source: {
    type: "api",
    url: "https://api.example.com/v1/data",
    headers: { "X-Api-Key": "..." },
    rateLimit: 200,      // ms delay between requests
    maxRetries: 3,       // retry failed requests
    paginate: {
      nextKey: "next_cursor",  // field in response for next page
      dataKey: "results",      // field containing the data array
    },
  },
});

Source Types

type SourceConfig = JsonSourceConfig | CsvSourceConfig | ApiSourceConfig;

interface JsonSourceConfig {
  type: "json";
  path?: string;      // local file path
  url?: string;       // remote URL
  rootKey?: string;   // key containing the data array
}

interface CsvSourceConfig {
  type: "csv";
  path?: string;
  url?: string;
  delimiter?: string;
  headers?: boolean;
}

interface ApiSourceConfig {
  type: "api";
  url: string;
  headers?: Record<string, string>;
  rateLimit?: number;     // ms between requests
  maxRetries?: number;
  paginate?: {
    nextKey: string;      // response field for next page token
    dataKey: string;      // response field containing data array
  };
}

Running Seeds

import { runSeed } from "@plantagoai/seeders";

const result = await runSeed(herbSeed, {
  dryRun: false,          // true = validate only, don't write
  batchSize: 500,         // docs per Firestore batch (default: 500)
  idempotent: true,       // skip docs that already exist (default: true)
  onProgress: (msg) => console.log(msg),
});

console.log(`Written: ${result.written}`);
console.log(`Skipped: ${result.skipped}`);
console.log(`Errors: ${result.errors.length}`);
for (const err of result.errors) {
  console.warn(err);
}

Pipeline

1. Fetch — Load data from source (JSON/CSV/API)
2. Transform — Apply transform function if provided
3. Validate — Check each record against Zod schema (invalid records logged and skipped)
4. Tag — Add _seed_tag field for cleanup tracking
5. Load — Batch-write to Firestore (500 ops per batch, auto-chunked)

SeedResult

interface SeedResult {
  written: number;
  skipped: number;
  errors: string[];
}

Cleaning Seeds

Remove all documents created by a specific seed:

import { cleanSeed } from "@plantagoai/seeders";

const deleted = await cleanSeed(herbSeed);
// Deletes all docs in "herbs" collection where _seed_tag == "herb-seed-v1"
console.log(`Deleted ${deleted} documents`);

Free Data Source Registry

Curated list of legally free, high-quality APIs and datasets. Each entry includes license verification, rate limits, and data freshness info.

import { FREE_DATA_SOURCES } from "@plantagoai/seeders";

Categories

FreeDataEntry

interface FreeDataEntry {
  url: string;           // API endpoint or dataset URL
  license: string;       // License type (CC0, MIT, public domain, etc.)
  description: string;   // What data is available
  format: "json" | "csv";
  rateLimit?: number;    // Requests per minute
}

Usage with defineSeed

import { defineSeed, FREE_DATA_SOURCES } from "@plantagoai/seeders";

const costOfLivingSeed = defineSeed({
  name: "cost-of-living",
  collection: "cost_of_living",
  tag: "col-seed-v1",
  schema: costOfLivingSchema,
  source: {
    type: "api",
    url: FREE_DATA_SOURCES.economics.costOfLiving.url,
    rateLimit: 500,
  },
  transform: (raw) => ({ ... }),
});

Project Usage