[source-shopify] doing a full refresh overwrite results in missing data

[ghenriqu]

Connector Name

source-shopify

Connector Version

2.4.24

What step the error happened?

During the sync

Relevant information

Using open source airbyte (version 0.63.19) with a connector from shopify (2.4.24) to bigquery (2.8.7). Whenever we run a 'full refresh | overwrite' sync for the product tables specifically the data doesn't refresh properly. We end up with a lot of missing data in our product table which causes product variants to be tied to id's that don't exist in our products table.

The only way that we've been able to see the data properly being refreshed is through this workaround:

• switching the sync mode from 'full refresh | overwrite' to 'incremental | append + deduped' and run a sync
• run another 'incremental | append + deduped' sync

Relevant log output

No response

Contribute

• Yes, I want to contribute