feat: MongoDB Atlas Serverless Support for Log-Based Replication

[anden-akkio]

This tap breaks for me with this error:

2023-10-27T14:57:35.899965Z [info     ] Environment 'staging' is active
2023-10-27T14:57:39.160144Z [warning  ] No state was found, complete import.
2023-10-27T14:57:41.504457Z [info     ] 2023-10-27 09:57:41,096 | INFO     | tap-mongodb          | Skipping deselected stream 'accounts'. cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.504641Z [info     ] 2023-10-27 09:57:41,096 | INFO     | tap-mongodb          | Beginning log_based sync of 'customers'... cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.504722Z [info     ] 2023-10-27 09:57:41,096 | INFO     | tap-mongodb          | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.504862Z [info     ] 2023-10-27 09:57:41,495 | CRITICAL | tap-mongodb          | operation_failure on collection.watch: $changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information, full error: {'ok': 0, 'errmsg': '$changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information', 'code': 8000, 'codeName': 'AtlasError'} cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.504936Z [info     ] 2023-10-27 09:57:41,495 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.3981499671936035, "tags": {"stream": "customers", "context": {}, "status": "failed"}} cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505001Z [info     ] 2023-10-27 09:57:41,495 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "customers", "context": {}}} cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505066Z [info     ] 2023-10-27 09:57:41,495 | ERROR    | tap-mongodb          | An unhandled error occurred while syncing 'customers' cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505127Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505192Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505254Z [info     ]     for _ in self._sync_records(context=context): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505323Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1073, in _sync_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505380Z [info     ]     for record_result in self.get_records(current_context): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505437Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/tap_mongodb/streams.py", line 283, in get_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505494Z [info     ]     raise operation_failure    cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505554Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/tap_mongodb/streams.py", line 253, in get_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505621Z [info     ]     change_stream = collection.watch(**change_stream_options) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505680Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/collection.py", line 2905, in watch cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505737Z [info     ]     return CollectionChangeStream( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505794Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 162, in __init__ cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505850Z [info     ]     self._cursor = self._create_cursor() cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505909Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 262, in _create_cursor cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.505975Z [info     ]     return self._run_aggregation_cmd(session=s, explicit_session=self._session is not None) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506035Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 256, in _run_aggregation_cmd cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506090Z [info     ]     return self._client._retryable_read( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506146Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506201Z [info     ]     return func(self, *args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506257Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1535, in _retryable_read cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506312Z [info     ]     return func(session, server, conn, read_pref) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506367Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/aggregation.py", line 164, in get_cursor cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506422Z [info     ]     result = conn.command(     cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506477Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/helpers.py", line 315, in inner cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506538Z [info     ]     return func(*args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506593Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/pool.py", line 932, in command cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506647Z [info     ]     return command(            cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506702Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/network.py", line 191, in command cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506757Z [info     ]     helpers._check_command_response( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506813Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/helpers.py", line 223, in _check_command_response cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506869Z [info     ]     raise OperationFailure(errmsg, code, response, max_wire_version) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506924Z [info     ] pymongo.errors.OperationFailure: $changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information, full error: {'ok': 0, 'errmsg': '$changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information', 'code': 8000, 'codeName': 'AtlasError'} cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.506979Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507039Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/bin/tap-mongodb", line 8, in <module> cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507097Z [info     ]     sys.exit(TapMongoDB.cli()) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507164Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507219Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507275Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507330Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507384Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507438Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507493Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507547Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507602Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 500, in invoke cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507657Z [info     ]     tap.sync_all()             cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507712Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 459, in sync_all cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507774Z [info     ]     stream.sync()              cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507830Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1186, in sync cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507889Z [info     ]     raise ex                   cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507943Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.507998Z [info     ]     for _ in self._sync_records(context=context): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508053Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1073, in _sync_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508108Z [info     ]     for record_result in self.get_records(current_context): cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508162Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/tap_mongodb/streams.py", line 283, in get_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508216Z [info     ]     raise operation_failure    cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508271Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/tap_mongodb/streams.py", line 253, in get_records cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508325Z [info     ]     change_stream = collection.watch(**change_stream_options) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508384Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/collection.py", line 2905, in watch cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508439Z [info     ]     return CollectionChangeStream( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508494Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 162, in __init__ cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508550Z [info     ]     self._cursor = self._create_cursor() cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508605Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 262, in _create_cursor cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508660Z [info     ]     return self._run_aggregation_cmd(session=s, explicit_session=self._session is not None) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508715Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/change_stream.py", line 256, in _run_aggregation_cmd cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508769Z [info     ]     return self._client._retryable_read( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508824Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508879Z [info     ]     return func(self, *args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508934Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1535, in _retryable_read cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.508994Z [info     ]     return func(session, server, conn, read_pref) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509049Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/aggregation.py", line 164, in get_cursor cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509104Z [info     ]     result = conn.command(     cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509159Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/helpers.py", line 315, in inner cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509213Z [info     ]     return func(*args, **kwargs) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509267Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/pool.py", line 932, in command cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509321Z [info     ]     return command(            cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509375Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/network.py", line 191, in command cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509429Z [info     ]     helpers._check_command_response( cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509483Z [info     ]   File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mongodb/venv/lib/python3.10/site-packages/pymongo/helpers.py", line 223, in _check_command_response cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509549Z [info     ]     raise OperationFailure(errmsg, code, response, max_wire_version) cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:41.509604Z [info     ] pymongo.errors.OperationFailure: $changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information, full error: {'ok': 0, 'errmsg': '$changeStream is not allowed or the syntax is incorrect, see the Atlas documentation for more information', 'code': 8000, 'codeName': 'AtlasError'} cmd_type=elb consumer=False name=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor producer=True stdio=stderr string_id=connection-hash-d3f7f4464283520e-table-specifier-hash-9bde77b80e0c14b6-extractor
2023-10-27T14:57:42.811960Z [info     ] Incremental state has been updated at 2023-10-27 14:57:42.811886.
2023-10-27T14:57:43.769236Z [error    ] Extractor failed
2023-10-27T14:57:43.769446Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Extractor failed

I've been able to narrow this down to the Change Stream API not being supported by MongoDB Atlas Serverless deployments (ref 1, ref 2).

We're attempting to use this in an Embedded ETL context to allow customers to stream data into our platform. However, as we don't know what the table itself will look like and we don't want to require customers to specify a replication key, INCREMENTAL replication mode is not a possibility. This leaves LOG_BASED as our only option, provided that FULL_TABLE isn't supported. Also, we want to support Serverless deployments, as not doing so likely prevents us from supporting a significant amount of our customers.

What are our options here, if we wish to use this tap instead of others? I saw the allow_modify_change_streams option, but that seems to just circumvent a permissions issue for the user, not fix the underlying database itself not supporting it.