Feat/Bulk and selective log deletion #25019

connermo · 2025-09-02T16:43:38Z

Summary

This PR implements comprehensive log clearing functionality for both chat and completion conversations. The implementation supports both bulk deletion (all logs) and selective deletion (specific conversation IDs).

Key Features:

Backend API endpoints for clearing chat/completion conversations
Frontend UI with "Clear All" and "Clear Selected" functionality
Multi-select checkboxes with confirmation dialogs

Technical Implementation:

Cascading deletion across database tables (conversations, messages, annotations, variables)
Background file cleanup with Celery tasks
Comprehensive localStorage clearing mechanism
SWR cache invalidation and cross-tab synchronization

Resolve #24626

Screenshots

Default shows "Clear All Logs" button only:

"Clear Selected" button shows when conversations are selected:

Confirmation dialog shows before execution:

Checklist

This change requires a documentation update, included: https://github.com/langgenius/dify-docs
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.
I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Additional Notes

Files Modified:

api/controllers/console/app/conversation.py - New DELETE endpoints
api/tasks/clean_uploaded_files_task.py - Background file cleanup task
web/app/components/app/log/filter.tsx - Clear buttons and confirmations
web/app/components/app/log/list.tsx - Multi-select functionality
web/app/components/app/log/index.tsx - Selection state management
web/service/log.ts - Enhanced clearing with 404 prevention
web/app/components/base/chat/chat-with-history/hooks.tsx - Error handling

Testing Completed:

Bulk deletion flow with explore page navigation
Selective deletion with remaining conversation accessibility
Error recovery scenarios (cross-tab conversation deletion)
localStorage synchronization across browser tabs

This PR introduces selective deletion capabilities for conversation logs, allowing users to delete specific conversations instead of only being able to clear all logs at once. ## Changes Made ### Backend API Updates - Enhanced DELETE endpoints in `api/controllers/console/app/conversation.py` to support selective deletion - Added `conversation_ids` parameter support for both chat and completion conversations - Maintained backward compatibility with existing "clear all" functionality - Added proper cleanup of related data (messages, files, annotations, etc.) ### Frontend UI Improvements - Added multi-selection checkboxes to log list interface - Implemented "Select All" functionality with proper state management - Enhanced filter component with "Clear Selected" button alongside existing "Clear All" - Added confirmation dialogs for both selective and full deletion operations - Fixed checkbox event handling to prevent conflicts with row clicks ### Internationalization - Added bilingual support (English/Chinese) for new UI elements - Included appropriate success/error messages for selective operations ## Key Features - **Multi-Selection**: Users can select individual conversations or use "Select All" - **Confirmation Dialogs**: Separate confirmations for clearing all vs clearing selected items - **Progress Feedback**: Loading states and success/error notifications - **Data Integrity**: Proper cleanup of associated files, messages, and metadata - **Backward Compatible**: Existing "Clear All" functionality remains unchanged ## Technical Details - Used React state management for selection tracking - Implemented proper event propagation handling for checkboxes - Enhanced API service layer with optional conversation ID parameters - Added comprehensive error handling and user feedback Fixes user pain point of having to delete all logs when only wanting to remove specific conversations.

@crazywoola

Following community feedback (@crazywoola), this refactors the delete functionality: ## Changes Made ### Service Layer Implementation - Added `ConversationService.clear_conversations()` method - Moved complex deletion logic from controllers to service layer - Added proper user validation and conversation ownership checks - Integrated with Celery task queue for async processing ### Celery Task Implementation - Created `clear_conversation_task.py` with batch processing - Handles large datasets efficiently with configurable batch sizes - Includes proper error handling and retry mechanisms - Comprehensive cleanup of files, messages, annotations, and related data - Progress tracking and detailed logging ### Controller Refactoring - Simplified DELETE endpoints to use service layer - Maintained backward compatibility with existing API - Added task tracking information in response - Proper error handling and user permissions ## Benefits - **Scalability**: Large datasets processed asynchronously - **Reliability**: Celery tasks with retry mechanisms - **Maintainability**: Clean separation of concerns - **Monitoring**: Progress tracking and detailed logging - **Performance**: Batch processing prevents memory issues ## Technical Details - Uses SQLAlchemy batch operations for efficiency - Proper transaction management with session handling - File cleanup integrated with storage layer - Comprehensive error handling for missing tables/data This addresses the architectural concern while maintaining all existing functionality.

- Fixed localStorage clearing mechanism to prevent 404 errors on explore pages - Added comprehensive error handling for conversation rename operations - Resolved merge conflicts and maintained clean code formatting - Enhanced conversation deletion to clear all conversation IDs to prevent app ID mismatches - Added proper error handling when conversations are deleted before rename operations complete - Fixed code style issues (line length, logging format, variable declaration order) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Resolved import conflicts in conversation.py and conversation_service.py - Maintained clear logs functionality with error prevention - Updated dependencies and imports to match upstream changes

Copilot

Pull Request Overview

This PR implements comprehensive log clearing functionality for both chat and completion conversations, supporting bulk deletion (all logs) and selective deletion (specific conversation IDs).

Backend API endpoints for clearing chat/completion conversations with cascading deletion across database tables
Frontend UI with "Clear All" and "Clear Selected" functionality including multi-select checkboxes and confirmation dialogs
Background file cleanup with Celery tasks and comprehensive localStorage clearing mechanism with SWR cache invalidation

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
web/service/log.ts	Added API functions for clearing conversations with localStorage and SWR cache management
web/i18n/zh-Hans/app-log.ts	Added Chinese translations for clear functionality
web/i18n/en-US/app-log.ts	Added English translations for clear functionality
web/app/components/base/chat/chat-with-history/hooks.tsx	Added error handling for deleted conversations and storage event listeners
web/app/components/app/log/list.tsx	Added multi-select checkboxes and selection state management
web/app/components/app/log/index.tsx	Added selection state management and prop passing
web/app/components/app/log/filter.tsx	Added clear buttons, confirmation dialogs, and clearing logic
api/tasks/clear_conversation_task.py	New Celery task for batch conversation deletion
api/tasks/clean_uploaded_files_task.py	New task for asynchronous file cleanup
api/services/conversation_service.py	Added clear conversations service method with error handling
api/controllers/console/app/conversation.py	Added DELETE endpoints for chat and completion conversations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-03T14:36:21Z

web/service/log.ts

+      // ADDITIONAL FIX: Also clear ALL conversation IDs to prevent explore page 404 errors
+      const keysToDelete = Object.keys(conversationIdInfo)
+      if (keysToDelete.length > 0) {
+        keysToDelete.forEach(key => {
+          delete conversationIdInfo[key]
+          console.log(`🧹 Cleared conversation ID for ${key} to prevent 404 errors`)
+        })
+        cleared = true
+      }


This code clears ALL conversation IDs from localStorage regardless of the app being cleared. This is overly aggressive and could affect other apps' conversation states. Consider only clearing conversation IDs for the specific app or related apps to avoid unintended side effects.

Suggested change

// ADDITIONAL FIX: Also clear ALL conversation IDs to prevent explore page 404 errors

const keysToDelete = Object.keys(conversationIdInfo)

if (keysToDelete.length > 0) {

keysToDelete.forEach(key => {

delete conversationIdInfo[key]

console.log(`🧹 Cleared conversation ID for ${key} to prevent 404 errors`)

})

cleared = true

}

// Only clear conversation ID for the current app to avoid affecting other apps

Copilot · 2025-09-03T14:36:21Z

web/service/log.ts

+export const clearCompletionConversations = async ({ appId, conversationIds }: { appId: string; conversationIds?: string[] }) => {
+  try {
+    const body = conversationIds ? { conversation_ids: conversationIds } : {}
+    const result = await del<any>(`/apps/${appId}/completion-conversations`, { body })
+
+    // Clear localStorage to prevent 404 errors on explore pages
+    if (typeof window !== 'undefined') {
+      const conversationIdInfo = JSON.parse(localStorage.getItem(CONVERSATION_ID_INFO) || '{}')
+
+      // Clear conversation ID for the current app (from logs page)
+      let cleared = false
+      if (conversationIdInfo[appId]) {
+        delete conversationIdInfo[appId]
+        cleared = true
+        console.log(`✅ Cleared conversation ID info for app ${appId}`)
+      }
+
+      // ADDITIONAL FIX: Also clear ALL conversation IDs to prevent explore page 404 errors
+      const keysToDelete = Object.keys(conversationIdInfo)
+      if (keysToDelete.length > 0) {
+        keysToDelete.forEach(key => {
+          delete conversationIdInfo[key]
+          console.log(`🧹 Cleared conversation ID for ${key} to prevent 404 errors`)
+        })
+        cleared = true
+      }
+
+      if (cleared) {
+        localStorage.setItem(CONVERSATION_ID_INFO, JSON.stringify(conversationIdInfo))
+        window.dispatchEvent(new StorageEvent('storage', {
+          key: CONVERSATION_ID_INFO,
+          newValue: JSON.stringify(conversationIdInfo),
+          storageArea: localStorage
+        }))
+      }
+    }


This code is duplicated from the clearChatConversations function. The localStorage clearing logic should be extracted into a shared utility function to avoid code duplication and ensure consistent behavior across both functions.

Suggested change

export const clearCompletionConversations = async ({ appId, conversationIds }: { appId: string; conversationIds?: string[] }) => {

try {

const body = conversationIds ? { conversation_ids: conversationIds } : {}

const result = await del<any>(`/apps/${appId}/completion-conversations`, { body })

// Clear localStorage to prevent 404 errors on explore pages

if (typeof window !== 'undefined') {

const conversationIdInfo = JSON.parse(localStorage.getItem(CONVERSATION_ID_INFO) || '{}')

// Clear conversation ID for the current app (from logs page)

let cleared = false

if (conversationIdInfo[appId]) {

delete conversationIdInfo[appId]

cleared = true

console.log(`✅ Cleared conversation ID info for app ${appId}`)

}

// ADDITIONAL FIX: Also clear ALL conversation IDs to prevent explore page 404 errors

const keysToDelete = Object.keys(conversationIdInfo)

if (keysToDelete.length > 0) {

keysToDelete.forEach(key => {

delete conversationIdInfo[key]

console.log(`🧹 Cleared conversation ID for ${key} to prevent 404 errors`)

})

cleared = true

}

if (cleared) {

localStorage.setItem(CONVERSATION_ID_INFO, JSON.stringify(conversationIdInfo))

window.dispatchEvent(new StorageEvent('storage', {

key: CONVERSATION_ID_INFO,

newValue: JSON.stringify(conversationIdInfo),

storageArea: localStorage

}))

}

}

// Utility function to clear conversation ID info from localStorage

function clearConversationIdInfoFromLocalStorage(appId: string) {

if (typeof window !== 'undefined') {

const conversationIdInfo = JSON.parse(localStorage.getItem(CONVERSATION_ID_INFO) || '{}')

// Clear conversation ID for the current app (from logs page)

let cleared = false

if (conversationIdInfo[appId]) {

delete conversationIdInfo[appId]

cleared = true

console.log(`✅ Cleared conversation ID info for app ${appId}`)

}

// ADDITIONAL FIX: Also clear ALL conversation IDs to prevent explore page 404 errors

const keysToDelete = Object.keys(conversationIdInfo)

if (keysToDelete.length > 0) {

keysToDelete.forEach(key => {

delete conversationIdInfo[key]

console.log(`🧹 Cleared conversation ID for ${key} to prevent 404 errors`)

})

cleared = true

}

if (cleared) {

localStorage.setItem(CONVERSATION_ID_INFO, JSON.stringify(conversationIdInfo))

window.dispatchEvent(new StorageEvent('storage', {

key: CONVERSATION_ID_INFO,

newValue: JSON.stringify(conversationIdInfo),

storageArea: localStorage

}))

}

}

}

export const clearCompletionConversations = async ({ appId, conversationIds }: { appId: string; conversationIds?: string[] }) => {

try {

const body = conversationIds ? { conversation_ids: conversationIds } : {}

const result = await del<any>(`/apps/${appId}/completion-conversations`, { body })

// Clear localStorage to prevent 404 errors on explore pages

clearConversationIdInfoFromLocalStorage(appId)

Copilot · 2025-09-03T14:36:21Z

api/controllers/console/app/conversation.py

+                try:
+                    db.session.query(MessageFeedback).where(MessageFeedback.message_id.in_(all_message_ids)).delete(
+                        synchronize_session=False
+                    )
+                except Exception:
+                    pass  # Table might not exist in this version


Using bare except Exception with pass statements throughout the deletion process silently ignores all errors, including serious ones like database connection issues. Consider catching specific exceptions (e.g., sqlalchemy.exc.ProgrammingError for missing tables) and logging warnings for unexpected errors to aid debugging.

Copilot · 2025-09-03T14:36:22Z

api/controllers/console/app/conversation.py

+            upload_file_ids = [mf.upload_file_id for mf in message_files if mf.upload_file_id]
+
+        # Delete all database records first (in transaction)
+        try:
+            # Delete message-related database records
+            if all_message_ids:


The same overly broad exception handling pattern is repeated in the chat conversations deletion method. This creates code duplication and the same maintenance issues as mentioned in the completion conversations method.

Copilot · 2025-09-03T14:36:22Z

web/app/components/app/log/list.tsx

+            <td className='w-10 whitespace-nowrap rounded-l-lg bg-background-section-burn pl-2 pr-1'>
+              <Checkbox
+                checked={isAllSelected}
+                indeterminate={isSomeSelected}
+                onCheck={handleSelectAll}


The header checkbox lacks an accessible label or aria-label to describe its purpose for screen readers. Consider adding an aria-label like "Select all conversations" to improve accessibility.

Suggested change

<td className='w-10 whitespace-nowrap rounded-l-lg bg-background-section-burn pl-2 pr-1'>

<Checkbox

checked={isAllSelected}

indeterminate={isSomeSelected}

onCheck={handleSelectAll}

onCheck={handleSelectAll}

aria-label={t('appLog.table.header.selectAllConversations')}

Copilot · 2025-09-03T14:36:22Z

web/app/components/app/log/list.tsx

+                  <Checkbox
+                    checked={selectedItems.includes(log.id)}
+                    onCheck={() => handleSelectItem(log.id)}


The individual row checkboxes lack accessible labels. Consider adding aria-label attributes that identify the specific conversation being selected, such as aria-label={Select conversation ${log.id}} to improve screen reader support.

Suggested change

<Checkbox

checked={selectedItems.includes(log.id)}

onCheck={() => handleSelectItem(log.id)}

onCheck={() => handleSelectItem(log.id)}

aria-label={`Select conversation ${log.id}`}

- Extract localStorage clearing logic into shared utility function (web/utils/localStorage.ts) - Add accessibility improvements with aria-labels for checkboxes - Improve exception handling with specific exception types (OperationalError, ProgrammingError) - Add comprehensive error logging for better debugging - Enhanced user permission validation with security logging This addresses the main feedback points from the PR review: - Code duplication reduction - Better error handling and debugging - Improved accessibility for screen readers - Enhanced security measures

- Change clearConversationIds to only clear specific app IDs by default - Add clearRelated option for handling related app IDs (app.id vs installedApp.id) - Add forceGlobalClear option for backwards compatibility (disabled by default) - Provide specialized functions for common use cases: - clearConversationIdsWithMapping() for log<->explore app ID mapping - smartClearConversationIds() for intelligent related ID detection - Update service calls to use safer, app-specific clearing - Add comprehensive logging to track what gets cleared This addresses the feedback: 'This code clears ALL conversation IDs from localStorage regardless of the app being cleared. This is overly aggressive and could affect other apps conversation states.' The new approach: - Default: Only clear the specific app ID provided - Optional: Clear related app IDs when explicitly specified - Safeguard: Prevent accidental clearing of unrelated apps

- Update import path from '@/config' to '@/app/components/base/chat/constants' - This resolves the build error: 'CONVERSATION_ID_INFO' is not exported from '@/config' The constant is properly defined in the chat constants file, matching other similar imports throughout the codebase.

Conner Mo and others added 3 commits August 28, 2025 14:01

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. 💪 enhancement New feature or request labels Sep 2, 2025

connermo mentioned this pull request Sep 2, 2025

feat: Add clear all logs functionality #24509

Closed

5 tasks

connermo changed the title ~~Feat/selective log deletion~~ Feat/complete and selective log deletion Sep 2, 2025

connermo changed the title ~~Feat/complete and selective log deletion~~ Feat/Bulk and selective log deletion Sep 2, 2025

Merge upstream main and resolve conflicts

642cd31

- Resolved import conflicts in conversation.py and conversation_service.py - Maintained clear logs functionality with error prevention - Updated dependencies and imports to match upstream changes

crazywoola requested review from laipz8200, JzoNgKVO and Copilot September 3, 2025 14:28

crazywoola self-assigned this Sep 3, 2025

Copilot AI reviewed Sep 3, 2025

View reviewed changes

Conner Mo added 2 commits September 4, 2025 22:46

Fix linter formatting issues

8518150

connermo force-pushed the feat/selective-log-deletion branch from f918b3a to 8518150 Compare September 4, 2025 14:47

Conner Mo added 2 commits September 4, 2025 22:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feat/Bulk and selective log deletion #25019

Feat/Bulk and selective log deletion #25019

connermo commented Sep 2, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

Uh oh!

Feat/Bulk and selective log deletion #25019

Are you sure you want to change the base?

Feat/Bulk and selective log deletion #25019

Conversation

connermo commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Features:

Technical Implementation:

Screenshots

Default shows "Clear All Logs" button only:

"Clear Selected" button shows when conversations are selected:

Confirmation dialog shows before execution:

Checklist

Additional Notes

Files Modified:

Testing Completed:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

connermo commented Sep 2, 2025 •

edited

Loading