core: fix fd memory leak in reactor_proc_add_fd, causing crash due to memory extinguish and too many open files

[danielzt]

Summary

rtpproxy: fix crash in timeout scenarios caused by file descriptor and memory leak, eventually leading to "Too many open files" errors and opensips crash.

Details

Under high load or unstable RTPProxy communication scenarios, OpenSIPS may trigger timeout conditions while handling RTPProxy sessions. In these cases, certain execution paths were not properly releasing allocated resources.

Specifically, when timeout or failure conditions occurred during RTPProxy interaction, associated file descriptors and memory structures were not consistently cleaned up. Over time, this resulted in:

• gradual file descriptor exhaustion (EMFILE / "Too many open files")
  lsof -p 2378548 | awk '{print $5}' | sort | uniq -c | sort -nr
  818 sock
  132 FIFO
  68 unix

• eventual crashes or service degradation

• a lot of logs generated, leading to full disk usage
  /sbin/opensips[2378548]: ERROR:rtpproxy:rtpproxy_io_new_callback: socket accept failed: Too many open files(24)

• increasing memory usage
  /sbin/opensips[2488108]: ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 16), please increase the "-M" command line parameter!
  /sbin/opensips[2488108]: ERROR:core:reactor_proc_add_fd: failed to allocate a reactor_proc callback

The issue is generic and not tied to a specific SIP endpoint or UAC implementation. It becomes visible mainly in high CPS environments or deployments where RTPProxy instances become temporarily unreachable or slow to respond.

This behavior was reproduced in production under sustained traffic and confirmed via monitoring tools (lsof, FD counters and memory observation).

Solution

The patch ensures proper cleanup of resources in all timeout and error execution paths inside the rtpproxy handling logic.

Main changes include:

• guaranteeing file descriptor release on timeout/failure branches
• fixing missing cleanup paths that previously bypassed resource deallocation
• ensuring consistent lifecycle management of allocated structures
• preventing descriptor accumulation during repeated timeout scenarios

The fix does not alter normal RTPProxy operation logic; it only corrects incomplete cleanup during exceptional flows.

Stress testing confirmed that file descriptor usage remains stable over long-running sessions after applying the patch.

Compatibility

This change is fully backward compatible:

• no configuration changes required
• no SIP signaling behavior changes
• no protocol or interoperability impact
• no module parameter changes

The patch only affects internal resource management during error handling.

Closing issues

N/A