{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":756620350,"defaultBranch":"main","name":"openhouse","ownerLogin":"linkedin","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2024-02-13T00:52:30.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/357098?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726788508.0","currentOid":""},"activityList":{"items":[{"before":"deccfe4a3511bfc4e7a4f6354f02bfbb0713bbc8","after":"cf33ab930166230cecfc60e6a6e7b58d4382d494","ref":"refs/heads/mor-feature-toggle","pushedAt":"2024-09-20T06:07:14.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"address comments","shortMessageHtmlLink":"address comments"}},{"before":"43275a82a34d34da8ecc390241e1de92efe5bd13","after":"d9f99ce944619c6899d727ac8620ce4acec538d0","ref":"refs/heads/main","pushedAt":"2024-09-19T23:06:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Support wildcard based matching for togglestatus endpoint (#203)\n\nThis PR contains logically two components: \r\n- The changes in\r\n`iceberg/openhouse/internalcatalog/src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalCatalog.java`\r\nis due to the discovery in the testing phase that the an inheritance\r\ndoesn't apply for annotation and AOP's proxy model requires the\r\nannotation to be applied on the object that got directly invoked. Thus\r\nthe annotation in this `createTable` is not necessary. Since adding\r\nannotation was the only reason why rewriting this method, removing this\r\nwhole method doesn't change any behavior.\r\n- The main part of this PR is to add wildcard matching for\r\n`hts/togglestatus` endpoint, so that for features like temporarily\r\nforbidding table-creation, we can simply insert one rule that contains\r\nwildcard for both tableId and databaseId.","shortMessageHtmlLink":"Support wildcard based matching for togglestatus endpoint (#203)"}},{"before":"0f251b5c51ff93ee2983b51ee34096e2a1ea8ac3","after":"43275a82a34d34da8ecc390241e1de92efe5bd13","ref":"refs/heads/main","pushedAt":"2024-09-19T21:39:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"sumedhsakdeo","name":"Sumedh Sakdeo","path":"/sumedhsakdeo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/773250?s=80&v=4"},"commit":{"message":"DLO: Multi-objective optimization for auto-compaction (#201)\n\n## Summary\r\n\r\nWe plan to develop Auto Compaction for for high ROI tables. Our plan is\r\nto treat this as a multi-objective optimization problem by aiming to\r\noptimizing two objectives -- maximize file count reduction and minimize\r\ncompute costs. We score and rank the tables, then choose the top-K\r\ntables for each iteration of compaction to remain under allocated\r\ncompute budget.\r\n\r\nThis will be used by the job scheduler for candidate selection.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [X] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [X] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [X] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"DLO: Multi-objective optimization for auto-compaction (#201)"}},{"before":null,"after":"48e3662fa5aaf669e0e0cde63432c35b99c4df32","ref":"refs/heads/toggle_wild_card","pushedAt":"2024-09-19T20:49:24.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Support wildcard matching for feature toggle","shortMessageHtmlLink":"Support wildcard matching for feature toggle"}},{"before":"b24c02b3af46df47e5325aba4cbd9e1885105979","after":"0f251b5c51ff93ee2983b51ee34096e2a1ea8ac3","ref":"refs/heads/main","pushedAt":"2024-09-18T00:44:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"rohitkum2506","name":"Rohit kumar","path":"/rohitkum2506","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1126602?s=80&v=4"},"commit":{"message":"Handle tblProperties for replication flow (#196)\n\n## Summary\r\n\r\nTable commits have strict checks for reserved table properties to avoid\r\nmodifying them. This leads to issues for use cases where cross cluster\r\niceberg commits for tables with same tableId should be compatible which\r\nallows features like table backup.\r\nThis PR introduces conditions when such check can be relaxed.\r\nSpecifically for cross cluster table replication.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [x] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [x] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Handle tblProperties for replication flow (#196)"}},{"before":"e6581f3e58ed8111c7030e4a0c705352f78f1155","after":"b24c02b3af46df47e5325aba4cbd9e1885105979","ref":"refs/heads/main","pushedAt":"2024-09-17T01:20:00.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"teamurko","name":"Stas Pak","path":"/teamurko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/270580?s=80&v=4"},"commit":{"message":"Add DLO strategies to shared table (#194)\n\n## Summary\r\nFor convenience this change also optionally allows to add strategy\r\ntraits to an output table. This allows to fetch all strategies for\r\nranking faster and more reliably compared to reading tblprops from all\r\ntables individually.\r\n\r\n[Issue](https://github.com/linkedin/openhouse/issues/#nnn)] Briefly\r\ndiscuss the summary of the changes made in this\r\npull request in 2-3 lines.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [ ] New Features\r\n- [x] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [x] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [ ] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nCreated test tables and run scheduler\r\n```\r\n...\r\n2024-09-16 16:58:03 INFO JobsScheduler:141 - Submitting and running 2 jobs based on the job type: DATA_LAYOUT_STRATEGY_GENERATION\r\n2024-09-16 16:58:03 INFO OperationTask:71 - Launching job for dbName: u_openhouse, tableName: test, creator: openhouse\r\n2024-09-16 16:58:03 INFO OperationTask:71 - Launching job for dbName: u_openhouse, tableName: dlo_strategies, creator: openhouse\r\n2024-09-16 16:58:11 INFO OperationTask:97 - Launched a job with id DATA_LAYOUT_STRATEGY_GENERATION_u_openhouse_test_d1c737be-f798-40ac-9a50-b3bf00fc426d for dbName: u_openhouse, tableName: test, creator: openhouse\r\n2024-09-16 16:58:11 INFO OperationTask:97 - Launched a job with id DATA_LAYOUT_STRATEGY_GENERATION_u_openhouse_dlo_strategies_978f260d-8450-47b9-b581-8ee4098ea3c4 for dbName: u_openhouse, tableName: dlo_strategies, creator: openhouse\r\n...\r\n2024-09-16 17:03:12 INFO OperationTask:143 - Finished job for entity dbName: u_openhouse, tableName: test, creator: openhouse: JobId DATA_LAYOUT_STRATEGY_GENERATION_u_openhouse_test_d1c737be-f798-40ac-9a50-b3bf00fc426d, executionId 0, runTime 45589, queuedTime 21762, state SUCCEEDED\r\n...\r\n```\r\nCheck dlo_strategies table contents\r\n```\r\nscala> spark.sql(\"select * from openhouse.u_openhouse.dlo_strategies\").show(200, false)\r\n+----------------+-----------------------+----------------------+------------------------------+----------------------+\r\n|fqtn |timestamp |estimated_compute_cost|estimated_file_count_reduction|file_size_entropy |\r\n+----------------+-----------------------+----------------------+------------------------------+----------------------+\r\n|u_openhouse.test|2024-09-16 23:42:30.677|0.5 |2.0 |2.77080972058891392E17|\r\n+----------------+-----------------------+----------------------+------------------------------+----------------------+\r\n\r\nscala> spark.sql(\"describe table openhouse.u_openhouse.dlo_strategies\").show(200, false)\r\n+------------------------------+---------------+-------+\r\n|col_name |data_type |comment|\r\n+------------------------------+---------------+-------+\r\n|fqtn |string | |\r\n|timestamp |timestamp | |\r\n|estimated_compute_cost |double | |\r\n|estimated_file_count_reduction|double | |\r\n|file_size_entropy |double | |\r\n| | | |\r\n|# Partitioning | | |\r\n|Part 0 |days(timestamp)| |\r\n+------------------------------+---------------+-------+\r\n```\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Add DLO strategies to shared table (#194)"}},{"before":"fcc64afa14683d1997cc793c6182470b510e8e64","after":"e6581f3e58ed8111c7030e4a0c705352f78f1155","ref":"refs/heads/main","pushedAt":"2024-09-16T20:23:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vikrambohra","name":"vikram bohra","path":"/vikrambohra","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24703997?s=80&v=4"},"commit":{"message":"Added a regex based db and table name storage selector (#191)\n\n## Summary\r\n\r\nIntroduced an implementation of Storage Selector that accepts a regex\r\nand storage-type and selects the provided storage type if db and table\r\nmatch the regex pattern. if pattern does not match default storage is\r\nused.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [x] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [x] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [x] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [x] Added new tests for the changes made.\r\n- [x] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n* Ran oh-s3-spark docker-compose with 2 configured storages (s3 & hdfs)\r\nwith default as s3 and hdfs for tables matching the regex\r\n\".*prod.*\\.table.*\"\r\n\r\n\r\n\r\n* Created 2 tables \r\n* \"database.table1\" that doesnt match regex and defaults to s3 storage\r\n* \"prod_database.table1\" that matches regex and chooses hdfs as storage\r\n\r\n\r\n* Verified selected storages in hts using mysql\r\n\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Added a regex based db and table name storage selector (#191)"}},{"before":"cf3d5bd16e260c511667c94243c91baa4cd7b57b","after":"e7dcdb1fdcc4e9b9753637b8baafd5ededc17658","ref":"refs/heads/dependabot/github_actions/github-actions-60c914ad0e","pushedAt":"2024-09-16T19:58:02.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump anothrNick/github-tag-action in the github-actions group\n\nBumps the github-actions group with 1 update: [anothrNick/github-tag-action](https://github.com/anothrnick/github-tag-action).\n\n\nUpdates `anothrNick/github-tag-action` from 1.69.0 to 1.70.0\n- [Release notes](https://github.com/anothrnick/github-tag-action/releases)\n- [Commits](https://github.com/anothrnick/github-tag-action/compare/1.69.0...1.70.0)\n\n---\nupdated-dependencies:\n- dependency-name: anothrNick/github-tag-action\n dependency-type: direct:production\n update-type: version-update:semver-minor\n dependency-group: github-actions\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump anothrNick/github-tag-action in the github-actions group"}},{"before":"80de42f1e7e33df33589fe249b4b91e4651c3706","after":"fcc64afa14683d1997cc793c6182470b510e8e64","ref":"refs/heads/main","pushedAt":"2024-09-13T01:00:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"maluchari","name":"Malini Mahalakshmi Venkatachari","path":"/maluchari","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1776785?s=80&v=4"},"commit":{"message":"Revert \"Add an option to enable/disable storage stats collection as part of TableStatsCollector\" (#193)\n\nReverts linkedin/openhouse#186\r\n\r\nReverting disabling storage stats completely. Will consider skipping\r\nthis only for certain tables or databases eventually.","shortMessageHtmlLink":"Revert \"Add an option to enable/disable storage stats collection as p…"}},{"before":null,"after":"09ee3e3672898c5969adf810499e05033e80b348","ref":"refs/heads/revert-186-malini/skip_storage_stats","pushedAt":"2024-09-13T00:31:23.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"maluchari","name":"Malini Mahalakshmi Venkatachari","path":"/maluchari","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1776785?s=80&v=4"},"commit":{"message":"Revert \"Add an option to enable/disable storage stats collection as part of T…\"\n\nThis reverts commit b37e2ac6b81a7b76f55034c79c51d38d2eb7764b.","shortMessageHtmlLink":"Revert \"Add an option to enable/disable storage stats collection as p…"}},{"before":"fbb61c4c2fc9fedce06fbd3a905354380ed6a50c","after":"80de42f1e7e33df33589fe249b4b91e4651c3706","ref":"refs/heads/main","pushedAt":"2024-09-12T21:20:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vikrambohra","name":"vikram bohra","path":"/vikrambohra","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24703997?s=80&v=4"},"commit":{"message":"Use storage selector in create table path and fileio read path (#187)\n\n## Summary\r\nPrimary changes\r\n1. Use storage returned by Storage selector to select storage for a new\r\ntable during creation\r\n2. Use correct FileIO for the storage type returned from HTS for a table\r\n3. Fixed bug where storage type always defaults to HDFS in hts\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [x] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [x] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [x] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [x] Added new tests for the changes made.\r\n- [x] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\nDocker compose S3 test, Db:d3, Table:t1\r\n1. Get table without creation -> Returns table not found with storage\r\nselected as s3 from storage selector\r\n\r\n\r\n\r\n2. Create table -> Returns success with storage selected as s3 from\r\nstorage selector, hts row with storageTyppe=s3 in hts row, minio s3 ui\r\nshows correct metadata\r\n\r\n\r\n\r\n\r\n\r\n3. Get table after table creation -> Returns success with storage\r\nselected as s3 from hts row\r\n\r\n\r\n\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Use storage selector in create table path and fileio read path (#187)"}},{"before":"9dd57af5087463ec598b13b2f5df52aa10c90322","after":"fbb61c4c2fc9fedce06fbd3a905354380ed6a50c","ref":"refs/heads/main","pushedAt":"2024-09-11T19:50:35.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"teamurko","name":"Stas Pak","path":"/teamurko","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/270580?s=80&v=4"},"commit":{"message":"Make leaf modules java17 forward compatible (#190)\n\n## Summary\r\nSome of the leaf modules can now build with java17, e.g.\r\ntables-test-fixtures and it's dependency modules. This is one several\r\nPRs that prepare for java17 switch. The changes are essentially\r\nnon-consequential for the consumers.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [ ] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n- [x] Upgrade\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [ ] Added new tests for the changes made.\r\n- [x] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Make leaf modules java17 forward compatible (#190)"}},{"before":"b37e2ac6b81a7b76f55034c79c51d38d2eb7764b","after":"9dd57af5087463ec598b13b2f5df52aa10c90322","ref":"refs/heads/main","pushedAt":"2024-09-11T18:58:16.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Feature Toggle Part III: Using autogen HTS client for feature-toggle (#174)\n\n## Summary\r\n\r\n\r\n\r\nThis PR completes `BaseTableToggleStatusesRepository` by injecting\r\n`ToggleStatusApi` so that it will connect to HTS to check the status of\r\na table's feature, defined by `featureId`, `tableId` and `databaseId`.\r\n\r\nMeanwhile, this PR also introduces the first use case for\r\nfeature-toggle: Disallow table creation when toggled off. See the change\r\nin `creationEligibilityCheck`. This change should have small to little\r\nimpact towards mysql-storage layer for the feature toggle, as there are\r\nonly <6K table-creation request (from which this code-path will be\r\ntriggered) in last 7d. Plus, this is a read-only operation for mysql.\r\n\r\n\r\n[Issue](https://github.com/linkedin/openhouse/issues/#nnn)] Briefly\r\ndiscuss the summary of the changes made in this\r\npull request in 2-3 lines.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [x] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [x] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [x] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nAdding unit tests cases for the new code added. \r\n\r\nAlso, tested the changes in the local environment, in the following\r\nsequence:\r\n```\r\nINSERT IGNORE INTO table_toggle_rule (feature, database_pattern, table_pattern, id, creation_time_ms) \\\r\n VALUES ('stop_create', 'demodb', 'demotable', DEFAULT, DEFAULT);\r\n\r\n mysql> select * from table_toggle_rule;\r\n+-------------+------------------+---------------+----+------------------+----------------------------+\r\n| feature | database_pattern | table_pattern | id | creation_time_ms | ETL_TS |\r\n+-------------+------------------+---------------+----+------------------+----------------------------+\r\n| demo | demodb | demotable | 1 | NULL | 2024-08-27 23:42:39.599344 |\r\n| stop_create | demodb | demotable | 2 | NULL | 2024-08-27 23:49:33.036207 |\r\n+-------------+------------------+---------------+----+------------------+----------------------------+\r\n```\r\n\r\nFirst step is ensuring the new feature-rule is inserted. \r\n\r\nThen, creating the tables: \r\n```\r\ncurl \"${curlArgs[@]}\" -XPOST http://localhost:8000/v1/databases/demodb/tables/ \\\r\n--data-raw '{\r\n \"tableId\": \"demotable\",\r\n \"databaseId\": \"demodb\",\r\n \"baseTableVersion\": \"INITIAL_VERSION\",\r\n \"clusterId\": \"LocalS3Cluster\",\r\n \"schema\": \"{\\\"type\\\": \\\"struct\\\", \\\"fields\\\": [{\\\"id\\\": 1,\\\"required\\\": true,\\\"name\\\": \\\"id\\\",\\\"type\\\": \\\"string\\\"},{\\\"id\\\": 2,\\\"required\\\": true,\\\"name\\\": \\\"name\\\",\\\"type\\\": \\\"string\\\"},{\\\"id\\\": 3,\\\"required\\\": true,\\\"name\\\": \\\"ts\\\",\\\"type\\\": \\\"timestamp\\\"}]}\",\r\n \"timePartitioning\": {\r\n \"columnName\": \"ts\",\r\n \"granularity\": \"HOUR\"\r\n },\r\n \"tableProperties\": {\r\n \"key\": \"value\"\r\n }\r\n}'\r\n```\r\n\r\nThe command above get the following responses: \r\n```\r\n{\"status\":\"FORBIDDEN\",\"error\":\"Forbidden\",\"message\":\"Feature [stop_create] has been toggled off for the [demodb.demotable]\",\"stacktrace\":null,\"cause\":\"Not Available\"}%\r\n```\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.\r\n\r\n---------\r\n\r\nCo-authored-by: Lei Sun ","shortMessageHtmlLink":"Feature Toggle Part III: Using autogen HTS client for feature-toggle (#…"}},{"before":"dd26bbb836e15c6cb14c8512ab89d445f4efb4b2","after":"deccfe4a3511bfc4e7a4f6354f02bfbb0713bbc8","ref":"refs/heads/mor-feature-toggle","pushedAt":"2024-09-10T03:52:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Fix unit test failure","shortMessageHtmlLink":"Fix unit test failure"}},{"before":null,"after":"dd26bbb836e15c6cb14c8512ab89d445f4efb4b2","ref":"refs/heads/mor-feature-toggle","pushedAt":"2024-09-10T01:30:00.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Feature-Toggle Use case: Allow preserved tablepropers to be modified on Table basis","shortMessageHtmlLink":"Feature-Toggle Use case: Allow preserved tablepropers to be modified …"}},{"before":"8861ba5fae520234e50ec40456186f1cd5e50bae","after":"cf3d5bd16e260c511667c94243c91baa4cd7b57b","ref":"refs/heads/dependabot/github_actions/github-actions-60c914ad0e","pushedAt":"2024-09-09T19:21:43.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump anothrNick/github-tag-action in the github-actions group\n\nBumps the github-actions group with 1 update: [anothrNick/github-tag-action](https://github.com/anothrnick/github-tag-action).\n\n\nUpdates `anothrNick/github-tag-action` from 1.69.0 to 1.70.0\n- [Release notes](https://github.com/anothrnick/github-tag-action/releases)\n- [Commits](https://github.com/anothrnick/github-tag-action/compare/1.69.0...1.70.0)\n\n---\nupdated-dependencies:\n- dependency-name: anothrNick/github-tag-action\n dependency-type: direct:production\n update-type: version-update:semver-minor\n dependency-group: github-actions\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump anothrNick/github-tag-action in the github-actions group"}},{"before":"896320ac142e3f35d778191166ffa1a67ad720ba","after":"b37e2ac6b81a7b76f55034c79c51d38d2eb7764b","ref":"refs/heads/main","pushedAt":"2024-09-09T01:42:52.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"maluchari","name":"Malini Mahalakshmi Venkatachari","path":"/maluchari","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1776785?s=80&v=4"},"commit":{"message":"Add an option to enable/disable storage stats collection as part of TableStatsCollector (#186)\n\n## Summary\r\n\r\nStorage stats computation can be expensive as this could involve\r\nrecursive directory walk in the file system. Hence making the storage\r\nstats computation optional so that this can be invoked as needed.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [X] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [ ] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [X] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Add an option to enable/disable storage stats collection as part of T…"}},{"before":"0bb178457e9a271f0a00e284638a00a1571bcf46","after":"896320ac142e3f35d778191166ffa1a67ad720ba","ref":"refs/heads/main","pushedAt":"2024-09-07T20:23:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"sumedhsakdeo","name":"Sumedh Sakdeo","path":"/sumedhsakdeo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/773250?s=80&v=4"},"commit":{"message":"Remove iceberg-azure v1.5.0 package loading (#188)","shortMessageHtmlLink":"Remove iceberg-azure v1.5.0 package loading (#188)"}},{"before":"0a0a179b9394fae3182b23d69d92231e8b3775a4","after":"90c13bbe51b9cad65ffcccee52b0d449d2e29c87","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-06T19:19:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Address comments","shortMessageHtmlLink":"Address comments"}},{"before":"e71bdc0837a49995f18b79a3c69b52b869e50173","after":"0a0a179b9394fae3182b23d69d92231e8b3775a4","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-06T18:17:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Add unit tests for featureGateAspect","shortMessageHtmlLink":"Add unit tests for featureGateAspect"}},{"before":"6ca1e883c8bb9bd3d19f3f782b486759ec82cc10","after":"e71bdc0837a49995f18b79a3c69b52b869e50173","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-06T00:31:59.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Fixing tests","shortMessageHtmlLink":"Fixing tests"}},{"before":"07e3dac6c95164ae14bddf34a304f4090be5c5e8","after":"6ca1e883c8bb9bd3d19f3f782b486759ec82cc10","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-05T23:23:20.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Realign on the package allocation for toggle classes","shortMessageHtmlLink":"Realign on the package allocation for toggle classes"}},{"before":"a35ba44462392a1e9d3007ad4a37e34b5e3581de","after":"07e3dac6c95164ae14bddf34a304f4090be5c5e8","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-04T22:19:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Fix unit test failure within internalcatalog module","shortMessageHtmlLink":"Fix unit test failure within internalcatalog module"}},{"before":"6a71e30bf50188dcb47a055791d6a222586832ee","after":"a35ba44462392a1e9d3007ad4a37e34b5e3581de","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-03T22:51:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"fix unit test in text-fixture module","shortMessageHtmlLink":"fix unit test in text-fixture module"}},{"before":"a31055e0a1a9259eb3602165867d77bb27667fd5","after":"6a71e30bf50188dcb47a055791d6a222586832ee","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-03T22:05:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Fix unit test failure","shortMessageHtmlLink":"Fix unit test failure"}},{"before":"9fc22c0b6b5b28b1e82c04cf8ec224edbdc53699","after":"a31055e0a1a9259eb3602165867d77bb27667fd5","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-03T21:08:00.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Remove unused code","shortMessageHtmlLink":"Remove unused code"}},{"before":"e68f6f60508851e2ce1a7d186c6c7bf96ef67f9a","after":"9fc22c0b6b5b28b1e82c04cf8ec224edbdc53699","ref":"refs/heads/feature_toggle_part3","pushedAt":"2024-09-03T21:06:27.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"autumnust","name":"Lei","path":"/autumnust","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5044213?s=80&v=4"},"commit":{"message":"Using annotation to enable feature toggle","shortMessageHtmlLink":"Using annotation to enable feature toggle"}},{"before":"508106274d4556ea64a11b4d6946f3a9b87850aa","after":"8861ba5fae520234e50ec40456186f1cd5e50bae","ref":"refs/heads/dependabot/github_actions/github-actions-60c914ad0e","pushedAt":"2024-09-02T19:44:24.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump anothrNick/github-tag-action in the github-actions group\n\nBumps the github-actions group with 1 update: [anothrNick/github-tag-action](https://github.com/anothrnick/github-tag-action).\n\n\nUpdates `anothrNick/github-tag-action` from 1.69.0 to 1.70.0\n- [Release notes](https://github.com/anothrnick/github-tag-action/releases)\n- [Commits](https://github.com/anothrnick/github-tag-action/compare/1.69.0...1.70.0)\n\n---\nupdated-dependencies:\n- dependency-name: anothrNick/github-tag-action\n dependency-type: direct:production\n update-type: version-update:semver-minor\n dependency-group: github-actions\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump anothrNick/github-tag-action in the github-actions group"}},{"before":"947c32472b725b8d073db85feab988c32afa30e7","after":"0bb178457e9a271f0a00e284638a00a1571bcf46","ref":"refs/heads/main","pushedAt":"2024-09-02T07:25:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"anjagruenheid","name":null,"path":"/anjagruenheid","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87397397?s=80&v=4"},"commit":{"message":"Empty set handling in DLO computation. (#180)\n\n## Summary\r\n\r\n\r\n\r\n[Issue](https://github.com/linkedin/openhouse/issues/#nnn)] Briefly\r\ndiscuss the summary of the changes made in this\r\npull request in 2-3 lines.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [x] Bug Fixes\r\n- [ ] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [ ] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [ ] Added new tests for the changes made.\r\n- [ ] Updated existing tests to reflect the changes made.\r\n- [x] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\nFor all the boxes checked, include a detailed description of the testing\r\ndone for the changes made in this pull request.\r\n\r\n# Additional Information\r\n\r\n- [ ] Breaking Changes\r\n- [ ] Deprecations\r\n- [ ] Large PR broken into smaller PRs, and PR plan linked in the\r\ndescription.\r\n\r\nFor all the boxes checked, include additional details of the changes\r\nmade in this pull request.","shortMessageHtmlLink":"Empty set handling in DLO computation. (#180)"}},{"before":"bd13635d7b8018960dd91b917cd4be50020e69f1","after":"947c32472b725b8d073db85feab988c32afa30e7","ref":"refs/heads/main","pushedAt":"2024-08-30T20:49:23.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"HotSushi","name":"Sushant Raikar","path":"/HotSushi","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6441597?s=80&v=4"},"commit":{"message":"Introduced StorageSelector interface to StorageProperties (#181)\n\n## Summary\r\n\r\nIntroduce table level storage selection.\r\n\r\n1. Introduced StorageSelector interface to allow the ability to select a\r\nstorage (Hdfs, S3 etc) at table level\r\n2. Added binding for it in StorageProperties. \r\n3. Added a default implementation \"DefaultStorageSelector\" that selects\r\nthe cluster storage for all tables.\r\n\r\n## Changes\r\n\r\n- [ ] Client-facing API Changes\r\n- [ ] Internal API Changes\r\n- [ ] Bug Fixes\r\n- [x] New Features\r\n- [ ] Performance Improvements\r\n- [ ] Code Style\r\n- [ ] Refactoring\r\n- [ ] Documentation\r\n- [x] Tests\r\n\r\nFor all the boxes checked, please include additional details of the\r\nchanges made in this pull request.\r\n\r\n## Testing Done\r\n\r\n- [ ] Manually Tested on local docker setup. Please include commands\r\nran, and their output.\r\n- [x] Added new tests for the changes made.\r\n- [x] Updated existing tests to reflect the changes made.\r\n- [ ] No tests added or updated. Please explain why. If unsure, please\r\nfeel free to ask for help.\r\n- [ ] Some other form of testing like staging or soak time in\r\nproduction. Please explain.\r\n\r\n1. Testing using yaml syntax in by adding a unit test in\r\nStoragePropertiesConfigTest\r\n2. Tested by mocks in a new test case StorageSelectorConfigTest","shortMessageHtmlLink":"Introduced StorageSelector interface to StorageProperties (#181)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEu5rrBAA","startCursor":null,"endCursor":null}},"title":"Activity · linkedin/openhouse"}