Skip to content

Commit 21ec99f

Browse files
nshestakovnshestakov
committed
Merge branch 'main' into PQ-move-partition
2 parents bb4a2b8 + 3f56c12 commit 21ec99f

File tree

325 files changed

+4560
-688
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

325 files changed

+4560
-688
lines changed

‎.github/scripts/telegram/README.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Telegram Integration Scripts
2+
3+
This directory contains scripts for integrating with Telegram for sending notifications and messages.
4+
5+
## Scripts
6+
7+
### 📨 `send_telegram_message.py`
8+
General-purpose script for sending messages or file contents to Telegram channels.
9+
10+
**Features:**
11+
- Send text messages or file contents
12+
- Support for message threads
13+
- Message chunking for large content
14+
- Disable web page previews
15+
- Configurable delays between messages
16+
17+
**Documentation:** [README_telegram.md](README_telegram.md)
18+
19+
### 👥 `parse_and_send_team_issues.py`
20+
Specialized script for parsing GitHub issues and sending team-specific notifications.
21+
22+
**Features:**
23+
- Parse team issues from formatted results
24+
- Send separate messages for each team
25+
- Support for multiple responsible users per team
26+
- Team lead mentions in messages
27+
- Support for message threads
28+
- Dry run mode for testing
29+
30+
**Documentation:** [README_team_issues.md](README_team_issues.md)
31+
32+
## Configuration
33+
34+
### Example Team Responsible Mapping
35+
See [team_leads_example.json](team_leads_example.json) for example configuration.
36+
37+
**Supported formats:**
38+
- Single responsible: `"team": "@team-lead"`
39+
- Multiple responsible: `"team": ["@lead1", "@lead2", "@lead3"]`
40+
41+
## GitHub Actions Setup
42+
43+
### 1. Secrets (Settings → Secrets and variables → Actions → Secrets)
44+
- `TELEGRAM_BOT_TOKEN` - your bot token
45+
- `TELEGRAM_YDBOT_TOKEN` - YDB bot token (used in workflow)
46+
47+
### 2. Variables (Settings → Secrets and variables → Actions → Variables)
48+
- `TELEGRAM_MUTE_CHAT_ID` - channel ID in format "chat_id/thread_id" (e.g., "2018419243/1")
49+
- `TEAM_TO_RESPONSIBLE_TG` - JSON string mapping teams to responsible users
50+
51+
### 3. Getting Bot Token
52+
1. Find [@BotFather](https://t.me/BotFather) in Telegram
53+
2. Send `/newbot`
54+
3. Follow instructions to create a bot
55+
4. Get token like `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`
56+
57+
### 4. Getting Channel ID
58+
For channel `https://t.me/c/2018419243/1`:
59+
- Channel ID: `-1002018419243`
60+
- Or use [@userinfobot](https://t.me/userinfobot) to get the ID
61+
62+
## Usage in GitHub Actions
63+
64+
These scripts are used in the `create_issues_for_muted_tests.yml` workflow to send notifications about muted tests to team channels.
65+
66+
## Requirements
67+
68+
- Python 3.7+
69+
- `requests` library
70+
- Valid Telegram bot token
71+
- Telegram chat/channel access

‎.github/scripts/telegram/README_team_issues.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Team Issues Parser and Sender
2+
3+
Script for parsing GitHub issues creation results and sending separate messages for each team to Telegram.
4+
5+
## Quick Start
6+
7+
```bash
8+
python .github/scripts/telegram/parse_and_send_team_issues.py \
9+
--file "created_issues.txt" \
10+
--bot-token "YOUR_BOT_TOKEN" \
11+
--team-channels '{"channels": {"main": "1234567890/1"}, "teams": {"team1": {"channel": "main", "responsible": ["@lead1"]}}, "default_channel": "main"}'
12+
```
13+
14+
## Parameters
15+
16+
- `--file` - Path to results file (required)
17+
- `--bot-token` - Telegram bot token (or TELEGRAM_BOT_TOKEN env var)
18+
- `--team-channels` - JSON string mapping teams to their channel configurations (or TEAM_CHANNELS env var) (required)
19+
- `--message-thread-id` - Thread ID for group messages (optional)
20+
- `--delay` - Delay between messages in seconds (default: 2)
21+
- `--dry-run` - Parse only without sending messages
22+
- `--max-retries` - Maximum number of retry attempts for failed messages (default: 5)
23+
- `--retry-delay` - Delay between retry attempts in seconds (default: 10)
24+
25+
## Message Format
26+
27+
```
28+
🔇 **07-09-24 new muted tests for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #team-name
29+
30+
fyi: @responsible1 @responsible2
31+
32+
- 🎯 [Issue URL](Issue URL) - `Issue Title`
33+
- 🎯 [Issue URL](Issue URL) - `Issue Title`
34+
35+
```
36+
37+
## Team Channels Configuration
38+
39+
```json
40+
{
41+
"channels": {
42+
"main": "1234567890/1",
43+
"secondary": "1234567890/5"
44+
},
45+
"teams": {
46+
"engineering": {
47+
"channel": "secondary",
48+
"responsible": ["@engineering-lead"]
49+
},
50+
"appteam": {
51+
"channel": "secondary",
52+
"responsible": ["@appteam-lead"]
53+
},
54+
"docs": {
55+
"channel": "main",
56+
"responsible": ["@docs-lead"]
57+
}
58+
},
59+
"default_channel": "main"
60+
}
61+
```
62+
63+
### Channel Selection Logic
64+
65+
1. **Team-specific channel**: If team exists in `teams` → use its `channel` from `channels`
66+
2. **Default channel**: If team not found → use `default_channel` from `channels`
67+
3. **Fallback**: If no `TEAM_CHANNELS` or `default_channel` → use `--chat-id` parameter
68+
69+
### Configuration Structure
70+
71+
- **`channels`** - Maps channel names to actual chat IDs
72+
- **`teams`** - Maps team names to their channel and responsible users
73+
- **`default_channel`** - Default channel name for teams not specified
74+
75+
### Responsible Users Logic
76+
77+
1. **From `teams[team].responsible`**: If team has `responsible` → use it
78+
2. **Empty**: If no responsible found → no responsible users in message
79+
80+
## Supported Input File Format
81+
82+
```
83+
🆕 **CREATED ISSUES**
84+
─────────────────────────────
85+
86+
👥 **TEAM** @ydb-platform/team1
87+
🎯 https://github.com/ydb-platform/ydb/issues/12345 - `Issue Title 1`
88+
🎯 https://github.com/ydb-platform/ydb/issues/12346 - `Issue Title 2`
89+
```
90+
91+
## Features
92+
93+
- ✅ Separate messages for each team
94+
- ✅ Markdown formatting with proper escaping
95+
- ✅ Responsible user mentions in messages
96+
- ✅ Message thread support
97+
- ✅ Dry run mode for testing
98+
- ✅ Automatic retry mechanism (5 retries with 10s delay by default)
99+
- ✅ Detailed error logging with failed message content
100+
- ✅ Automatic chat_id/thread_id parsing from "2018419243/1" format

‎.github/scripts/telegram/README_telegram.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Telegram Message Sender
2+
3+
Universal script for sending messages to Telegram channels.
4+
5+
## Quick Start
6+
7+
```bash
8+
# Send file (automatically detected)
9+
python .github/scripts/telegram/send_telegram_message.py \
10+
--message "path/to/file.txt" \
11+
--bot-token "YOUR_BOT_TOKEN" \
12+
--chat-id "1234567890"
13+
14+
# Send direct message
15+
python .github/scripts/telegram/send_telegram_message.py \
16+
--message "Hello, World!" \
17+
--bot-token "YOUR_BOT_TOKEN" \
18+
--chat-id "1234567890"
19+
```
20+
21+
## Parameters
22+
23+
- `--bot-token` - Telegram bot token (required)
24+
- `--chat-id` - Chat/channel ID (required)
25+
- `--message` - Message text or path to file to send (automatically detected)
26+
- `--parse-mode` - Parse mode (Markdown, HTML, None) - default: Markdown
27+
- `--delay` - Delay between messages in seconds - default: 1
28+
- `--max-length` - Maximum message length - default: 4000
29+
- `--message-thread-id` - Thread ID for group messages (optional)
30+
- `--disable-web-page-preview` - Disable link previews (default: True)
31+
- `--max-retries` - Maximum number of retry attempts for failed messages (default: 5)
32+
- `--retry-delay` - Delay between retry attempts in seconds (default: 10)
33+
34+
## Features
35+
36+
- ✅ Automatic splitting of long messages into chunks
37+
- ✅ Markdown formatting support
38+
- ✅ Delay between messages to avoid API limits
39+
- ✅ Message thread support
40+
- ✅ Disable link previews
41+
- ✅ Automatic retry mechanism (5 retries with 10s delay by default)
42+
- ✅ Detailed error logging with failed message content
43+
- ✅ Error handling with detailed diagnostics

‎.github/scripts/telegram/data_examples/message.txt

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
2+
🔒 **CLOSED ISSUES**
3+
─────────────────────────────
4+
5+
✅ **Closed** https://github.com/ydb-platform/ydb/issues/21591
6+
📝 **Unmuted tests:**
7+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[11]`
8+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[12]`
9+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[14]`
10+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[15]`
11+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[17]`
12+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[4]`
13+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[6]`
14+
15+
✅ **Closed** https://github.com/ydb-platform/ydb/issues/22871
16+
📝 **Unmuted tests:**
17+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[10]`
18+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[13]`
19+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[16]`
20+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[18]`
21+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[19]`
22+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[1]`
23+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[20]`
24+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[21]`
25+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[22]`
26+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[2]`
27+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[3]`
28+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[7]`
29+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[8]`
30+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[9]`
31+
32+
✅ **Closed** https://github.com/ydb-platform/ydb/issues/23878
33+
📝 **Unmuted tests:**
34+
• `ydb/tests/compatibility/s3_backups/test_export_import_s3.py.TestExportImportS3.test_tables[mixed_main_and_prestable-25-2]`
35+
• `ydb/tests/compatibility/s3_backups/test_export_import_s3.py.TestExportImportS3.test_tables[mixed_prestable-25-3_and_prestable-25-2]`
36+
37+
✅ **Closed** https://github.com/ydb-platform/ydb/issues/23928
38+
📝 **Unmuted tests:**
39+
• `ydb/core/tablet_flat/ut/Vacuum.StartVacuumWithFollowers`
40+
41+
✅ **Closed** https://github.com/ydb-platform/ydb/issues/23972
42+
📝 **Unmuted tests:**
43+
• `ydb/tests/datashard/async_replication/test_async_replication.py.TestAsyncReplication.test_async_replication[table_index_4__SYNC-pk_types0-all_types0-index0---SYNC]`
44+
45+
─────────────────────────────
46+
47+
🔓 **PARTIALLY UNMUTED ISSUES**
48+
─────────────────────────────
49+
50+
⚠️ **Partially unmuted** https://github.com/ydb-platform/ydb/issues/21765
51+
📝 **Unmuted tests:**
52+
• `ydb/tests/functional/tpc/large/test_tpch_spilling.py.TestTpchSpillingS10.test_tpch[5]`
53+
54+
⚠️ **Partially unmuted** https://github.com/ydb-platform/ydb/issues/23874
55+
📝 **Unmuted tests:**
56+
• `ydb/tests/functional/tpc/large/py3test.[test_tpcds.py] chunk`
57+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1.test_tpcds[39]`
58+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1.test_tpcds[43]`
59+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1DecimalNative.test_tpcds[20]`
60+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1DecimalNative.test_tpcds[22]`
61+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1DecimalNative.test_tpcds[33]`
62+
• `ydb/tests/functional/tpc/large/test_tpcds.py.TestTpcdsS0_1DecimalNative.test_tpcds[72]`
63+
64+
⚠️ **Partially unmuted** https://github.com/ydb-platform/ydb/issues/23877
65+
📝 **Unmuted tests:**
66+
• `ydb/tests/compatibility/test_compatibility.py.TestCompatibility.test_simple[restart_main_to_main-row]`
67+
• `ydb/tests/compatibility/test_compatibility.py.TestCompatibility.test_simple[restart_prestable-25-2_to_main-row]`
68+
• `ydb/tests/compatibility/test_ctas.py.TestCTASOperations.test_ctas_olap[rolling_prestable-25-2_to_main]`
69+
• `ydb/tests/compatibility/test_ctas.py.TestCTASOperations.test_ctas_olap[rolling_prestable-25-2_to_prestable-25-3]`
70+
• `ydb/tests/compatibility/test_ctas.py.TestCTASOperations.test_ctas_olap[rolling_stable-25-1-4_to_prestable-25-2]`
71+
• `ydb/tests/compatibility/test_ctas.py.TestCTASOperations.test_ctas_oltp[rolling_prestable-25-2_to_main]`
72+
• `ydb/tests/compatibility/test_ctas.py.TestCTASOperations.test_ctas_oltp[rolling_stable-25-1-4_to_prestable-25-2]`
73+
• `ydb/tests/compatibility/test_simple_reader.py.TestSimpleReaderMixedCluster.test_simple_reader_mixed_cluster[mixed_prestable-25-3_and_prestable-25-2]`
74+
• `ydb/tests/compatibility/test_simple_reader.py.TestSimpleReaderTabletTransfer.test_simple_reader_tablet_transfer[rolling_prestable-25-2_to_main]`
75+
• `ydb/tests/compatibility/test_simple_reader.py.TestSimpleReaderTabletTransfer.test_simple_reader_tablet_transfer[rolling_prestable-25-2_to_prestable-25-3]`
76+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_log[mixed_main_and_prestable-25-2-column]`
77+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_tpch1[mixed_main_and_prestable-25-2-column-date64]`
78+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_tpch1[mixed_main_and_prestable-25-2-column]`
79+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_tpch1[mixed_main_and_prestable-25-2-row]`
80+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_tpch1[mixed_prestable-25-3_and_prestable-25-2-column-date64]`
81+
• `ydb/tests/compatibility/test_stress.py.TestStress.test_tpch1[mixed_prestable-25-3_and_prestable-25-2-column]`
82+
• `ydb/tests/compatibility/test_system_views.py.TestSystemViewsRegistry.test_domain_sys_dir[restart_main_to_prestable-25-2]`
83+
• `ydb/tests/compatibility/test_system_views.py.TestSystemViewsRollingUpgrade.test_path_resolving[rolling_prestable-25-2_to_main]`
84+
• `ydb/tests/compatibility/test_table_schema_compatibility.py.TestTableSchemaCompatibilityRolling.test_schema_changes_during_rolling_upgrade[rolling_prestable-25-2_to_main]`
85+
• `ydb/tests/compatibility/test_table_schema_compatibility.py.TestTableSchemaCompatibilityRolling.test_table_operations_during_rolling_upgrade[rolling_prestable-25-2_to_main]`
86+
• `ydb/tests/compatibility/test_topic.py.TestTopicRollingUpdate.test_write_and_read[rolling_prestable-25-2_to_main]`
87+
88+
─────────────────────────────
89+
90+
🆕 **CREATED ISSUES**
91+
─────────────────────────────
92+
93+
👥 **TEAM** @ydb-platform/Unknown
94+
https://github.com/orgs/ydb-platform/teams/Unknown
95+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
96+
🎯 https://github.com/ydb-platform/ydb/issues/24322 - `Mute ydb/tests/functional/restarts/test_restarts.py.TestRestartSingleBlock42.test_restart_single_node_is_ok in main`
97+
98+
99+
👥 **TEAM** @ydb-platform/appteam
100+
https://github.com/orgs/ydb-platform/teams/appteam
101+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
102+
🎯 https://github.com/ydb-platform/ydb/issues/24315 - `Mute ydb/public/sdk/cpp/src/client/topic/ut/with_direct_read_ut/TxUsage.Transactions_Conflict_On_SeqNo_Table in main`
103+
🎯 https://github.com/ydb-platform/ydb/issues/24317 - `Mute ydb/public/sdk/cpp/tests/integration/sessions_pool/YdbSdkSessionsPool1Session.CustomPlan/0 in main`
104+
105+
106+
👥 **TEAM** @ydb-platform/engineering
107+
https://github.com/orgs/ydb-platform/teams/engineering
108+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
109+
🎯 https://github.com/ydb-platform/ydb/issues/24318 - `Mute ydb/tests/compatibility 40 tests in main`
110+
🎯 https://github.com/ydb-platform/ydb/issues/24319 - `Mute ydb/tests/compatibility 13 tests in main`
111+
🎯 https://github.com/ydb-platform/ydb/issues/24320 - `Mute ydb/tests/compatibility/s3_backups 9 tests in main`
112+
🎯 https://github.com/ydb-platform/ydb/issues/24321 - `Mute ydb/tests/compatibility/olap 7 tests in main`
113+
114+
115+
👥 **TEAM** @ydb-platform/fq
116+
https://github.com/orgs/ydb-platform/teams/fq
117+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
118+
🎯 https://github.com/ydb-platform/ydb/issues/24313 - `Mute ydb/tests/fq/pq_async_io/ut/TDqPqReadActorTest.LoadCorruptedState in main`
119+
🎯 https://github.com/ydb-platform/ydb/issues/24314 - `Mute ydb/core/fq/libs/row_dispatcher/format_handler/ut 2 tests in main`
120+
121+
122+
👥 **TEAM** @ydb-platform/qp
123+
https://github.com/orgs/ydb-platform/teams/qp
124+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
125+
🎯 https://github.com/ydb-platform/ydb/issues/24312 - `Mute ydb/core/kqp/ut/service/KqpService.CloseSessionsWithLoad in main`
126+
127+
128+
👥 **TEAM** @ydb-platform/system-infra
129+
https://github.com/orgs/ydb-platform/teams/system-infra
130+
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
131+
🎯 https://github.com/ydb-platform/ydb/issues/24316 - `Mute ydb/core/tx/schemeshard/ut_vector_index_build_reboots/VectorIndexBuildTestReboots.BaseCase+Prefixed[PipeResets] in main`

‎.github/scripts/telegram/data_examples/team_leads_example.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"Unknown": "@unknownresponsible",
3+
"appteam": "@appteamresponsible",
4+
"engineering": ["@engineeringlead1", "@engineeringlead2"],
5+
"fq": "@fqresponsible",
6+
"qp": ["@qplead", "@qptechlead"],
7+
"system-infra": "@systeminfraresponsible",
8+
"platform": ["@platformlead", "@platformarchitect"],
9+
"storage": "@storageresponsible",
10+
"analytics": "@analyticsresponsible",
11+
"security": ["@securitylead", "@securityengineer"],
12+
"devops": "@devopsresponsible",
13+
"qa": ["@qalead", "@qaautomationlead"],
14+
"infrastructure": "@infrastructureresponsible"
15+
}

0 commit comments

Comments
 (0)