Usually i was setting something like this
wasuser soft nofile 65535
wasuser hard nofile 65535
where wasuser is the service WAS user (non root) and 65535 is the number of open file written in ST wiki for single server on single VM.
In last of my Sametime 9 test environment , with several servers on same VM , i was getting some insufficent resource starting more than 2 server at same time.
Inside this IBM technote i've found other setting required on Linux forA WAS server from 6.1 to 8.5.5 so i've added inside /etc/security/limits.conf the following setting
wasuser soft nproc 131072
wasuser hard nproc 131072
and after that all service was able to start using wasuser
The nproc setting could be useful also for IBM Connections or Websphere Portal (in case you have OutOfMemory issue) because this setting is also for WAS 8.0