Sync facts

• Running an initial sync from your DB to Oso Cloud.
• Keeping facts up-to-date in production with Oso Sync.
• Configuration for PostgreSQL, MongoDB, and CSV.
• Limitations and best practices.

​

Sync facts in production

1. Decide how to represent authorization data as facts.
2. Run a one-time initial sync to populate Oso Cloud.
3. Keep facts in sync using dual writes and periodic reconciliation.

​

Initial sync with Oso Sync

oso-cloud reconcile --perform-updates reconcile.yaml

​

Configure Oso Sync

​

PostgreSQL

version: 1
source: postgres
facts:
  has_relation(Repository:_, String:parent, Organization:_):
    db: app_db
    query: |-
      select repository.public_id, organization.public_id
      from repository
      join organization
        on organization.id = repository.organization_id
dbs:
  app_db:
    connection_string: postgresql://oso:oso@host:5432/dbname

• version: optional, defaults to 1.
• source: optional, defaults to postgres.
• facts:
  • Fact type uses positional variables (_) that map to query columns in order.
  • db matches an entry in dbs.
  • query returns all facts of that type.
  • Example: has_relation(Repository:_, String:parent, Organization:_) has variables in the first and third arguments. repository.public_id fills the first argument (Repository), and organization.public_id fills the third argument (Organization).
• dbs:
  • Maps unique names to database connection details.
  • connection_string is a PostgreSQL connection URI or an environment variable: connection_string: $ENV_VAR_NAME.

​

MongoDB

version: 1
source: mongodb
facts:
  has_relation(Repository:_, String:parent, Organization:_):
    db: app_db
    collection: has_relation
    fields:
      - name: repository
      - name: organization
        is_array: true
    query:
      find: {}
dbs:
  app_db:
    connection_string: mongodb://oso:oso@somemongo.instance.aws.com:27017/dbname

• version: must be 1.
• source: must be mongodb.
• facts:
  • Fact type uses positional variables (_) that map to query columns in order.
  • collection: name containing fact data.
  • fields: maps to positional arguments in fact type.
    • At most one field can have is_array: true (automatically unwound).
  • query can be either:
  • find: standard find query.
  • aggregate: aggregation pipeline (cannot use $out).
  • Example: has_relation(Repository:_, String:parent, Organization:_) has variables in the first and third arguments.
• dbs:
  • connection_string must be a valid MongoDB connection URI or an environment variable: connection_string: $ENV_VAR_NAME.

​

Comma-separated Values (CSV)

version: 1
source: csv
facts:
  has_relation(Repository:_, String:parent, Organization:_):
    fields:
      - name: repository
      - name: organization
    path: /path/to/has_relation.csv

• version: must be 1.
• source: must be csv.
• facts:
  • Fact type uses positional variables (_) that map to query columns in order.
  • fields: must match the CSV header exactly.
    • Order must match positional arguments in the fact type.
  • path: local path to CSV file.
  • Example: has_relation(Repository:_, String:parent, Organization:_) has variables in the first and third arguments.

​

Oso Sync limitations

1. At most one Oso Sync command should be run at a time for a given environment. If multiple Oso Sync commands are run in parallel for an environment, you may see HTTP 429 errors.
2. The maximum size of the application data per fact type is 10GB. To synchronize larger data sets, you may consider “sharding” a single fact type across multiple fact type definitions in the YAML configuration by substituting a concrete value for one or more of the arguments. Before:
   Copy

   Ask AI

   has_relation(Repository:_, String:_, Organization:_): ...
   

   After:
   Copy

   Ask AI

   has_relation(Repository:_, String:parent, Organization:_): ...
   has_relation(Repository:_, String:child, Organization:_): ...
   

3. The diff may include transient false positives due to our comparing a point-in-time snapshot of your database to Oso Cloud, which continues to receive changes. Transient false positives should not appear on successive invocations of Oso Sync and do not indicate issues with how your application updates facts in Oso Cloud.

​

Docker

FROM public.ecr.aws/osohq/reconcile:latest
ARG CONFIG_PATH
RUN test -n "$CONFIG_PATH" || (echo "CONFIG_PATH argument must be set to path of your reconcile.yaml" && false)
WORKDIR /app
COPY $CONFIG_PATH /app/config.yaml
ENTRYPOINT ["/app/reconcile", "reconcile", "/app/config.yaml"]