groupby errors out

[rootsmusic]

I'm testing a new environment by running a .ipynb file that ran before without errors (but the modin version may have been slightly older). Now, my notebook errors out on the line (below) after 10 minutes.
df_by_quarter = df.groupby(['quarter']).sum().reset_index()

Is this a new bug caused by modin? If yes, I won't be able to open an issue because I don't have a reproducible example. But there are several issues about groupby.

Error Log

2023-10-22 16:20:00,655 - distributed.scheduler - WARNING - Worker failed to heartbeat within 300 seconds. Closing: <WorkerState 'tcp://127.0.0.1:54098', name: 4, status: running, memory: 44, processing: 7>
2023-10-22 16:20:00,659 - distributed.scheduler - WARNING - Worker failed to heartbeat within 300 seconds. Closing: <WorkerState 'tcp://127.0.0.1:54106', name: 1, status: running, memory: 48, processing: 5>
2023-10-22 16:20:00,675 - distributed.scheduler - WARNING - Communication with worker tcp://127.0.0.1:54106 failed during replication: CommClosedError: in <TCP (closed) ConnectionPool.gather local=tcp://127.0.0.1:54131 remote=tcp://127.0.0.1:54106>: Stream is closed
2023-10-22 16:20:00,691 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:54097 failed to acquire keys: ('function-3561ea653abf486ab236a859625dd3a9',)
2023-10-22 16:20:00,782 - distributed.scheduler - ERROR - Error transitioning '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' from 'waiting' to 'processing'
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' processing>
2023-10-22 16:20:01,094 - distributed.core - ERROR - <TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' processing>
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' processing>
2023-10-22 16:20:01,120 - distributed.core - ERROR - Exception while handling op register-client
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 968, in _handle_comm
result = await result
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 5532, in add_client
await self.handle_stream(comm=comm, extra={"client": client})
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 1049, in handle_stream
await handler(**merge(extra, msg))
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' forgotten>
Task exception was never retrieved
future: <Task finished name='Task-181' coro=<Server._handle_comm() done, defined at c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py:874> exception=AssertionError(<TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' forgotten>)>
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 968, in _handle_comm
result = await result
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 5532, in add_client
await self.handle_stream(comm=comm, extra={"client": client})
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 1049, in handle_stream
await handler(**merge(extra, msg))
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-cad16f6b-751d-4cd1-a304-b5e2aa359495' forgotten>
2023-10-22 16:20:18,887 - distributed.nanny - WARNING - Worker process still alive after 3.1999975585937506 seconds, killing
2023-10-22 16:20:18,887 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-22 16:23:30,321 - distributed.scheduler - ERROR - Error transitioning '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' from 'waiting' to 'processing'
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' processing>
2023-10-22 16:23:30,321 - distributed.core - ERROR - <TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' processing>
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' processing>
2023-10-22 16:23:30,375 - distributed.core - ERROR - Exception while handling op register-client
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 968, in _handle_comm
result = await result
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 5532, in add_client
await self.handle_stream(comm=comm, extra={"client": client})
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 1049, in handle_stream
await handler(**merge(extra, msg))
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' forgotten>
Task exception was never retrieved
future: <Task finished name='Task-214885' coro=<Server._handle_comm() done, defined at c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py:874> exception=AssertionError(<TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' forgotten>)>
Traceback (most recent call last):
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 968, in _handle_comm
result = await result
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 5532, in add_client
await self.handle_stream(comm=comm, extra={"client": client})
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\core.py", line 1049, in handle_stream
await handler(**merge(extra, msg))
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4645, in update_graph
self._create_taskstate_from_graph(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 4583, in _create_taskstate_from_graph
self.transitions(recommendations, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 7639, in transitions
self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2040, in _transitions
new_recs, new_cmsgs, new_wmsgs = self._transition(key, finish, stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 1934, in _transition
recommendations, client_msgs, worker_msgs = func(
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 2304, in transition_waiting_processing
return self._add_to_processing(ts, ws, stimulus_id=stimulus_id)
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3208, in _add_to_processing
return {}, {}, {ws.address: [self._task_to_msg(ts)]}
File "c:\Users\roots.conda\envs\modin\lib\site-packages\distributed\scheduler.py", line 3375, in _task_to_msg
assert ts.priority, ts
AssertionError: <TaskState '_deploy_dask_func-b592d31a-d604-4e97-ab72-b4402eafeb0d' forgotten>
2023-10-22 16:30:00,663 - distributed.scheduler - WARNING - Worker failed to heartbeat within 300 seconds. Closing: <WorkerState 'tcp://127.0.0.1:54109', name: 3, status: running, memory: 0, processing: 0>

Installed Versions

UserWarning: Setuptools is replacing distutils.

INSTALLED VERSIONS

commit : 4c01f64
python : 3.10.11.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 151 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

Modin dependencies

modin : 0.24.1
ray : None
dask : 2023.10.0
distributed : 2023.10.0
hdk : None

pandas dependencies

pandas : 2.1.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : 0.29.35
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.16.1
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.10.0
gcsfs : None
matplotlib : 3.7.3
numba : 0.57.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[Garra1980]

Are you using dask engine here?

[rootsmusic]

Yes @Garra1980, I should have specified. (I'm currently creating another environment to test again.)

[rootsmusic]

Sorry, I created another environment and chose to use the Ray engine, which didn't output any error.