RSSChangelog

This change is relevant for users of the index.json and catalog.json metadata. API and yente users as well as those users who download static files based on location are not affected.

As the number of datasets in the default collections grows, the metadata size is becoming a factor: the main index file is now larger than two megabytes. In order to prevent scaling issues in the future, we're splitting up the dataset metadata into two files: index.json and statistics.json. index.json will contain fewer summary facts about each dataset. The extended statistics (e.g. the number of entities in the dataset linked to each country) will be in the more in-depth statistics.json.

The following fields will be removed from dataset metadata:

• schemata
• properties
• targets (and all nested items)
• things (and all nested items)

These fields continue to be published in a statistics.json file. A statistics.json file is published with each data export of a dataset, and referenced from the index.json via the statistics_url field.

Additionally, the following fields will be removed in favor of replacements that are already in index.json:

• sources - use datasets instead
• externals - use datasets instead

The followthemoney data model currently stores the citizenship of individuals in the nationality property. After being advised the the two concepts are not identical in some jurisdictions, we've now also introduced a citizenship property. From the effective date we will begin moving country affiliations for individuals in the citizenship property if that nomenclature is used in the data source (e.g. the UK sanctions list).

Data consumers should check both properties in the future. To get a complete picture of the countries linked to an individual you may also want to check the birthCountry and country field. The latter serves as a catch-all field for affiliations that may not involve citizenship or holding a passport - simple residence might be enough.

See: Person schema.

The sanctions regime under Section 353 of the Corrupt and Undemocratic Actors Report has expired in December 2023. To reflect this, we've removed the topic sanction from all entities in the dataset. We will remove the relevant dataset entirely after November 1, 2024, and move the formerly-designated entities for reference to the US Special Legislative Exclusions dataset

Section 353 concerns foreign individuals who have been reported for knowingly engaging in activities that undermine democratic processes or institutions, participating in significant corruption, or obstructing investigations into such acts in El Salvador, Guatemala, Honduras, and Nicaragua.

The permId property for LSEG/Refinitiv company codes has been moved up from the Company to the Organization schema to enable reflecting government entities (using the PublicBody schema) also receiving these identifiers.

yente 4.1.0 (release page) is a minor patch release. It introduces clearer error reporting during index runs, and fixes the number of matches reported in the total section of the /match API. Various dependencies have been updated to their latest version, including followthemoney.

A soft length limit in Unicode codepoints has been added for all properties. These can be seen in the data dictionary. The goal of this is to make it easier for data consumers to import our data into systems with fixed-length column types.

Property values are not yet guaranteed to be limited to this value, but our tooling now alerts us when values are longer than this, so that we can identify sources which don’t adhere to sensible limits and eventually enforce hard limits.

Imposing a length limit has also identified many instances where the data required further cleaning, which we've implemented as needed.