If you are a system administrator that is responsible for managing AIX systems that run SAP, then you’ve probably had an experience similar to the following?

 

OK, so one day my SAP Basis administrator contacts me and says “I can’t start saposcol......can you please reboot the system?” I quickly reply, “Was there an error message when trying to restart saposcol?”. He replies, “No”. Again I return very quickly, “OK, have you checked to see if there are any shared memory segments left for saposcol?”.  Just as quick, he replies “How do I do that?”

 

Together we try starting saposcol and what we find is that it thinks it’s already running (as shown below, PID 327924). But there is no such process!

 

$ saposcol -l

-l

09:37:24 22.01.2009   LOG: Effective User Id is root

***********************************************************************

* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

* Usage:  saposcol -l: Start OS Collector

*         saposcol -k: Stop  OS Collector

*         saposcol -d: OS Collector Dialog Mode

*         saposcol -s: OS Collector Status

* The OS Collector (PID 327924) is already running .....

************************************************************************

$ ps -fp 327924

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

So we try to stop saposcol. This of course fails, as there is no PID 327924 running!

 

$ saposcol -k

 

Setting Stop Flag :

09:19:29 22.01.2009   LOG: ==== Stop Flag was set by saposcol (kill_collector()).

 

09:19:29 22.01.2009   LOG: ====  The collection process will stop as soon as possible

 

********************

can't kill process 327924.

kill: No such process

ERROR:No reaction from collecting process 327924.

Please kill collecting process.

 

My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?

 

Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.

 

http://www.saptechies.com/os-collector-saposcol/

7. Question: How do I remove the shared memory key of saposcol?

Answer: Sometimes it may be necessary to remove the shared memory key of saposcol (see point 6). Caution: Please be very careful! This procedure should be performed only after checking that saposcol is really not running (see point 4) and only in cases when other options (see point 6) really do not work! For this, execute command “ipcs -ma” and note the line that contains saposcol key “4dbe”. You need the shared memory ID. After that, execute command ipcrm -m ID. Now the command “saposcol -s” should show that saposcol is not running and that the shared memory is not attached. The shared memory key will be created automatically by the saposcol when the collector is next started: “saposcol -l”.

 

So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.

 

There are several process id’s ‘attached’ to this segment. However, only one of them actually exists (PID 2293794).

 

# ipcs -ma | grep 4dbe

m   2097156 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      0   1870188 467164 2293794  9:39:43  9:39:43  8:12:43

 

# ps -fp 1870188

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 467164

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 2293794

     UID     PID    PPID   C    STIME    TTY  TIME CMD

  sapadm 2293794       1   0 19:26:41      -  1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01

 

I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.

 

# ipcrm -m 2097156

# ipcs -ma | grep 4dbe

 

We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.

 

# ipcs -ma | grep 4dbe

m   3145732 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      1   1870188 2293818 2207892 10:14:25 10:14:25 10:12:47

 

# ps -ef | grep oscol

  sapadm 2064616       1   0 10:12:53      -  0:00 saposcol -l

   

# /usr/sap/SAP/SYS/exe/run/saposcol -s

**************************************************************

Collector Versions :

  running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

  dialog  : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

Shared Memory       : attached

Number of records   : 17640

Active Flag         : active (01)

Operating System    : AIX aix01 3 5 00C01C704C00

Collector PID       : 2064616 (001F80E8)

Collector           : running

Start time coll.    : Thu Jan 22 10:12:53 2009

 

Current Time        : Thu Jan 22 10:17:46 2009

 

Last write access   : Thu Jan 22 10:17:38 2009

 

Last Read  Access   : Thu Jan 22 10:16:16 2009

 

Collection Interval : 10 sec (next delay).

Collection Interval : 10 sec (last ).

Status              : free

Collect Details     : required

Refresh             : required

 

Header Extention Structure

Number of x-header      Records : 1

Number of Communication Records : 60

Number of free Com.     Records : 60

Resulting offset to 1.data rec. : 61

 

Trace level             : 2

 

Collector in IDLE - mode ? : NO

  become idle after 300 sec without read access.

  Length of Idle Interval  : 60 sec

  Length of norm.Interval  : 10 sec

**************************************************************

 

Problem solved and no reboot required.....this is AIX after all!  :)

 

Tags:  aix chris gibson ipcrm memory ipcs saposcol shared