PyAudio distorted recording when while loop too busy

I have a Python script that monitors an audio stream in real time and uses a moving average to determine when there is audio and set start and stop points based on when it is above or below a given threshold. Because the script runs 24/7, I avoid excessive memory usage by removing part of the audio stream after about 4 hours of audio.

def record(Jn):
  global current_levels
  global current_lens
  device_name = j_lookup[Jn]['device']
  device_index = get_index_by_name(device_name)
  audio = pyaudio.PyAudio()
  stream = audio.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, input_device_index=device_index, frames_per_buffer=CHUNK)

  recorded_frames = []
  quantized_history = []
  long_window = int(LONG_MOV_AVG_SECS*RATE/CHUNK) # converting seconds to while loop counter

  avg_counter_to_activate_long_threshold = LONG_THRESH*long_window
  safety_window = 1.5*avg_counter_to_activate_long_threshold
  long_thresh_met = 0

  long_start_selection = 0
  while True:
    data = stream.read(CHUNK, exception_on_overflow=False)
    recorded_frames.append(data)
    frame_data = struct.unpack(str(CHUNK) + 'h', data)
    frame_data = np.array(frame_data)
    sum_abs_frame = np.sum(np.abs(frame_data))
    quantized_history.append(0 if sum_abs_frame < j_lookup[Jn]['NOISE_FLOOR'] else 1)
    current_levels[Jn] = sum_abs_frame

    counter = len(recorded_frames)
    current_lens[Jn] = counter
    if counter >= long_window:
      long_movavg = sum(quantized_history[counter-long_window:counter])/long_window
      if long_movavg >= LONG_THRESH and long_thresh_met != 1:
        long_start_selection = int(max(counter - safety_window, 0))
        long_thresh_met = 1
      if long_movavg < LONG_THRESH and long_thresh_met == 1:
        long_end = int(counter)
        long_thresh_met = 2
        save_to_disk(recorded_frames[long_start_selection:long_end], audio, Jn)

    if counter > MAX_LOOKBACK_PERIOD: # don't keep endless audio history to avoid excessive memory usage
      del recorded_frames[0]
      del quantized_history[0]
      long_start_selection = max(0, long_start_selection - 1) # since you deleted first element, the recording start index is now one less

What I have above works, but what I noticed is that once I hit the four hour mark (the if counter > MAX_LOOKBACK_PERIOD statement at the very end becomes true), any audio saved after that point starts to sound distorted. For example, before the four hour point, the audio looks like:

after the four hour mark, it looks like:

You can see the distortion appearing as these vertical spikes on the spectrogram. I assume the del function is just taking so long that the while loop can't keep up with the audio stream and this is somehow causing the distortion, but I'm not sure. It has to be related to del somehow because the distortion only appears once the if counter > MAX_LOOKBACK_PERIOD becomes true.

Any idea how to address this?

from PyAudio distorted recording when while loop too busy