A Flatfile plugin that provides functionality to find and remove duplicate records from a sheet based on specified field values or custom logic.
dedupePlugin
function takes two parameters:
Option | Type | Default | Description |
---|---|---|---|
on | string | undefined | The field key (e.g., ‘email’) to check for duplicate values. Required when using the keep option. |
keep | ’first’ | ‘last’ | undefined | Determines which record to keep when a duplicate is found. ‘first’ keeps the first record encountered, ‘last’ keeps the last record encountered. |
custom | function | undefined | A custom function that receives a batch of records and a set of unique values. Should return an array of record IDs to be deleted. Overrides the keep option. |
debug | boolean | false | When set to true , the plugin may output additional logging for development and debugging purposes. |
opts
object, the plugin will not perform any deduplication. You must provide either the keep
and on
options or a custom
function for the plugin to work. If the keep
option is used, the on
option becomes mandatory.
jobOperation
string in your listener code and the operation
value in your Sheet configuration’s action. They must be identical.on
option exists in your Sheet configuration and is spelled correctly.on
field.custom
function, add logging to debug its logic.on
option: Error: \
on` is required when `keep` is first`Error: Field "non_existent_field" not found
Error: Dedupe must be called from a sheet-level action
operation
property of this action must exactly match the jobOperation
string passed to the dedupePlugin
function.keep: 'last'
option may not function as expected on very large datasets where duplicate records are spread across different pages of data. The keep: 'first'
option is generally more reliable for large datasets as it correctly tracks unique values across all pages. For a reliable “keep last” implementation on large datasets, a custom
function should be used.jobHandler
utility, which provides standardized job management. Any error thrown during the dedupe function’s execution will be caught, and the job will be marked as ‘failed’ with the corresponding error message. The plugin also performs its own configuration checks and will throw descriptive errors for common misconfigurations.