Hi Wilbert,

Thanks for your reply.I have seen first hand it removing tables under SYSTEM user as well as changing privileges and such. WE have restored once an SPS7 rev 70 on SPS9 rev 92 and some of the default system privileges were revoked and had to be re-granted for instance for the hana cockpit.

Then again when we restored Hana 92 on 112 with XS Advanced enabled, all XS Advanced default schemas were gone as well as a few others that were added in SPS11. It actually killed XS Advanced when restored on it. The service was red

Yes my colleagues were saying the same thing about i never had an issue as well but I don't believe it anymore having seen these issues pop like this even on default base installation before I add any of my own things

Wilbert Jeuken wrote:

 

You will have the revision 93 database content running on HANA with 112 binaries. I don't see a problem with that right away. You might get issues when you subsequently want to update this 112 to another revision, as then you'll get the post processing for the new revision on your restored data.

This reads as if the recovered backup from a SPS 9 version would run with SPS 11 binaries in some sort of SPS 9-mode.

That's not the case.

In fact upon restart, SAP HANA finds that this is an old version of a SAP HANA database and performs the required catalog upgrade procedures automatically.

What it doesn't automatically do is the import of of the delivery units that come with each revision.

So these will need to be manually imported - just like the upgrade would do - in order to get all pieces correctly working on the new SPS.

It should be clear, that the recovery-and-upgrade approach is not meant to be the main upgrade procedure. Instead, in-place upgrades are what is mainly expected and documented.

Thanks Krishna and Raj for replying.

I have to calculate WTD, MTD and so on...

I see the Scalar Procedures for MTD, but for YTD, WTD, QTD I dont have. Can you please share any details on the code.

and if I want to create a Calc Column - WTD,what formula( expression) I should use. I have CALWEEK( from M_TIME_DIMENSION) and Date from VBAK

SDQ

As per the understanding I have about the functionality of SDQ, it performs the data cleansing and validation of data such as duplicate check etc.

Can SDQ handle UI applications error message handling(During validation).

For Eg: Using a FIORI application we are trying to create a Vendor number which already exists in the Database. This check is done at SDQ level and creation will be blocked, will this validation failure reflect as a message automatically or it has to interpreted using the SQL error code numbers.

Regards,

Murthy

Adding to above question by Murthy,

How does SDQ support application level data quality like address validation and duplicate checks, do we still need to use data services DQM. Does SDQ offer this feature if no, how can we achieve this functionality in HANA environment.

Can you please provide some information or point to any reference document which describes these features.

Hello Saurabh,

did you only grant select access or did you grant select access with grant option._SYS_REPO needs select access rights with grant option to grant the select privilege to your user.

For more details please have a look to the documentation (Required Permissions - SAP HANA Modeling Guide - SAP Library) or use the search function here to find the other questions already answered to that topic.

Regards,

Florian

Thanks I indeed missed to include with grant option.

Regards

Saurabh

HI experts,

thank you for your suggestions. So what i understand is that as a basis administartor

better to go with HANASUP and HANATECH. So what kind of HA pdf i have to read fir this.

will HA100 and HA200 be sufficient?

Regards,

Srujay

Hello Srujay,

On training.sap.com page you will find the certification codes and there in topic areas of each the percentages and the course codes are given. Starting from the higher percentaged topics cover all books or guides to be ready.

C_HANATEC151 - SAP Certified Technology Associate - SAP HANA (Edition 2015)

C_HANASUP_1 - SAP Certified Support Associate - SAP HANA

Regards,

Yuksel AKCINAr

Hi folks,

In SAP source system we have what we call a firefighter role where we can assign access so that a developer for example could have temporary access to production in order to troubleshoot a problem.  All firefighter access has an expiration date and is logged for auditor purposes.  I'm not aware of a way to do this within HANA studio security.  I thought about perhaps setting up special roles and then setting up audit on that role to keep a trail of who was granted access although it doesn't resolve the expiration issue.  I'm curious how have others handled this in HANA that may need the same type of access auditing/restriction?

Thanks!

-Patrick

The following concepts are important to understand emergency access management(firefight):

Firefighter: the user who requires emergency access

Firefighter ID: the user ID with elevated privileges.

Firefighting: the act of using a Firefighter ID to perform tasks in an emergency

Owner: the user responsible for a Firefighter ID and the assignment of controllers

and Firefighters

Controller: the user who reviews and approves (if required) the log files generated

from firefighting activities

Centralized Firefighting: using the GRC system as the centralized console

through which Firefighters can logon to different system for firefighting

Decentralized Firefighting: Firefighters can directly logon to the plug-in systems

for firefighting; using the GRC system only for maintaining emergency access

assignments and reporting

The above definitions are well explained and courtesy to the below book:

SAP Access Control 10.0 SP10

Hope it answers your query

Hi Aparajit, I understand the Firefighter concept as well as GRC which we have here however I'm talking about from HANA Studio security perspective.  I don't think any of this works with HANA security although please correct me if I'm wrong.  I searched through the document for mention of HANA security and found nothing.

Thanks,

-Patrick

I confirm what Ning Tong said.

Also during our updating from rev 85 to 112 , we faced this warning, but after the manually re-import from HANA studio, all went fine.

Hi Experts,

How to find the version of a SAP HANA system. Great if you share the menu path used in SAP screen to find the HANA system version.

Thanks

Sakthi

You could use PowerDesigner for this kind of thing.

However, most SAP applications don't implement foreign key constraints so in terms of discovering relationships your chances may be slim, regardless of the tool used.

Hi Michael

These are the dumps OOM Files generated.

Indeed, there was a previous webdispatcher OOM trace file with MEMORY_OOM Information

=========================================

Runtime dump protocol for PID 1272

Rundirectory: /usr/sap/HED/HDB10/uyclxxx.mdp.local

Process start time: 2016-04-14 10:23:10 791 Local

Runtime dump time: 2016-04-25 16:17:30 291 Local

SectionTimeout: 30sec

KillTimeout: 300sec

Runtime Dump Filter: (OOM Runtime dump filter)

=========================================

Table of contents:

  [BUILD]  Build information

  [STACK_SHORT]  Short call stacks and pending exceptions of all threads

  [PROCESS_INFO]  Process Info

  [OS_MEMORY]  Operating system information about memory

  [THREADS]  Running threads

  [MEMORY_OOM]  Information about current out of memory situation

[BUILD]  Build information: (2016-04-25 16:17:30 291 Local)

Version      : 1.00.102.01.1444147999 (fa/newdb100_rel)

Build host    : ld7272

Build time    : 2015-10-06 19:16:10

Platform      : linuxx86_64

Compiler      : gcc (SAP release 20150922, based on SUSE gcc47-4.7.2_20130108-0.15.45.6763.0.PTF.871552) 4.7.2 20130108 [gcc-4_7-branch revision 195014]

Branch        : fa/newdb100_rel

Git hash      : 319000a825a2a9e19fd235302bd80b9a4ff8bb62

Git mergetime : 2015-10-06 18:13:19

Weekstone    : 0000.00.0

[OK]

--

[STACK_SHORT]  Short call stacks and pending exceptions of all threads: (2016-04-25 16:17:30 291 Local)

3082495[thr=3170]: Pool is inactive

--

2256[thr=1307]: WebDispatcherWorkerThread0007 is inactive

--

2257[thr=1308]: WebDispatcherWorkerThread0008 is inactive

--

2258[thr=1309]: WebDispatcherWorkerThread0009 is inactive

--

2259[thr=1310]: WebDispatcherWatchdogThread is inactive

--

2285[thr=1311]: JobexWatchdog is inactive

--

2287[thr=1312]: JobexMainDispatcher is inactive

--

2289[thr=1313]: JobWrkDiag is inactive

--

2291[thr=1314]: JobWrk0002 is inactive

--

2293[thr=1315]: JobWrk0003 is inactive

--

2306[thr=1318]: SelfWatchDog at

1: 0x00007fd561f51199 in syscall+0x15 (libc.so.6)

2: 0x00007fd563679ecc in Synchronization::Semaphore::timedWait(unsigned long, unsigned long, Execution::Context&)+0x1f8 at LinuxFutexOps.hpp:53 (libhdbbasis.so)

3: 0x00007fd563590a34 in MemoryManager::GlobalMemoryHandler::pmWait()+0x30 at Semaphore.hpp:255 (libhdbbasis.so)

4: 0x00007fd563590b3b in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLock(bool, unsigned long, unsigned long, bool&)+0xc7 at GlobalMemoryHandler.cpp:6835 (libhdbbasis.so)

5: 0x00007fd563590d8c in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemoryHlp(MemoryManager::IpmmMasterScope&, unsigned long)+0x88 at GlobalMemoryHandler.cpp:6804 (libhdbbasis.so)

6: 0x00007fd563590eb8 in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemory(MemoryManager::IpmmMasterScope&, ltt::auto_object<Synchronization::LockedScope<Synchronization::NamedProcessMutexNoAlloc, true> >&, unsigned long, bool)+0xe4 at GlobalMemoryHandler.cpp:6762 (libhdbbasis.so)

7: 0x00007fd5635a015c in MemoryManager::GlobalMemoryHandler::provideMemoryAndReturnSize(bool, bool, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x698 at GlobalMemoryHandler.cpp:4603 (libhdbbasis.so)

8: 0x00007fd5635a0cd5 in MemoryManager::GlobalMemoryHandler::provideMemory(unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x41 at GlobalMemoryHandler.cpp:4387 (libhdbbasis.so)

9: 0x00007fd56353fa45 in MemoryManager::AllocatorConfigurationSPI::provideMemory(unsigned long, unsigned long, bool, char const*, unsigned long)+0x21 at AllocatorConfigurationImpl.cpp:2765 (libhdbbasis.so)

10: 0x00007fd56360c0d1 in MemoryManager::VirtualAllocTracer::reserveMemory(unsigned long&, bool, bool, void*, unsigned long)+0x290 at VirtualAllocTracer.cpp:187 (libhdbbasis.so)

11: 0x00007fd5635ade6c in MemoryManager::mmapOverride(void*, unsigned long, int, int, int, long, void*)+0x2b8 at MallocProxy.cpp:1660 (libhdbbasis.so)

12: 0x00007fd561ed22ec in _IO_file_doallocate_internal+0x78 (libc.so.6)

13: 0x00007fd561edf7ac in _IO_doallocbuf_internal+0x28 (libc.so.6)

14: 0x00007fd561eddd64 in _IO_new_file_underflow+0x1b0 (libc.so.6)

15: 0x00007fd561edf7ee in _IO_default_uflow_internal+0xa (libc.so.6)

16: 0x00007fd561ec6ca0 in _IO_vfscanf_internal+0x6060 (libc.so.6)

17: 0x00007fd561ecf5e8 in __fscanf+0x84 (libc.so.6)

18: 0x00007fd56471a5a0 in checkLockFile()+0x80 at SelfWatchdog.cpp:328 (libhdbbasement.so)

19: 0x00007fd56471ab96 in TrexService::SelfWatchdog::checkLockFile()+0x12 at SelfWatchdog.cpp:126 (libhdbbasement.so)

20: 0x00007fd56471ad7e in TrexService::SelfWatchdog::run(void*)+0x11a at SelfWatchdog.cpp:113 (libhdbbasement.so)

21: 0x00007fd564746c80 in TrexThreads::PoolThread::run()+0xb10 at PoolThread.cpp:294 (libhdbbasement.so)

22: 0x00007fd564748e20 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:136 (libhdbbasement.so)

23: 0x00007fd563471f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

24: 0x00007fd563472dfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

--

2313[thr=1319]: Pool is inactive

--

2351[thr=1324]: JobWrk0004 is inactive

--

2806[thr=1718]: Pool is inactive

--

212384[thr=22075]: JobWrk0005 is inactive

--

470852[thr=11482]: Pool is inactive

--

1430006[thr=16810]: Pool is inactive

--

5[thr=0]: <SYSTEM> is inactive

--

638[thr=1286]: TimerThread is inactive

--

667[thr=1289]: TraceSegmentCompressor at

1: 0x00007fd5633e94e2 in Execution::ContextFunctions::printStack(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool)+0xc0 at ContextFunctions.cpp:628 (libhdbbasis.so)

2: 0x00007fd5632a415e in Diagnose::RuntimeDumpSectionStackShort::dumpContext(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::RuntimeDumpTimer&, Diagnose::FaultDescriptor*)+0xfa at CrashDump.cpp:959 (libhdbbasis.so)

3: 0x00007fd5632a6b38 in Diagnose::RuntimeDumpSectionStackShort::dump(ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::RuntimeDumpTimer&, Diagnose::FaultDescriptor*)+0x784 at CrashDump.cpp:982 (libhdbbasis.so)

4: 0x00007fd5632a3a44 in Diagnose::RuntimeDumpHelper::writeSectionNoSEH(Diagnose::RuntimeDumpSectionBase&, Diagnose::InternalRuntimeDumpTimer&)+0x180 at CrashDump.cpp:534 (libhdbbasis.so)

5: 0x00007fd56329ea62 in Diagnose::CrashDump::executeInternal(ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::FaultDescriptor*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*, char const*)+0x690 at CrashDump.cpp:1928 (libhdbbasis.so)

6: 0x00007fd5632a00aa in Diagnose::CrashDump::executeWithFilename(char const*, Diagnose::FaultDescriptor*, char const*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*)+0xd6 at CrashDump.cpp:1662 (libhdbbasis.so)

7: 0x00007fd5632a056a in Diagnose::RuntimeDump::createSystemDump(char const*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*)+0x46 at CrashDump.cpp:2325 (libhdbbasis.so)

8: 0x00007fd5635e1214 in MemoryManager::reportOOM(unsigned long)+0x2d0 at ReportMemoryProblems.cpp:1063 (libhdbbasis.so)

9: 0x00007fd5635f488c in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, unsigned long, bool)+0xb8 at PoolAllocator.cpp:777 (libhdbbasis.so)

10: 0x00007fd5635f55c0 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x250 at PoolAllocator.cpp:1449 (libhdbbasis.so)

11: 0x00007fd561f14651 in __alloc_dir+0x60 (libc.so.6)

12: 0x00007fd5636c02b4 in System::UX::opendir(char const*)+0x20 at SystemCallsUNIX.cpp:75 (libhdbbasis.so)

13: 0x00007fd5634cdde8 in FileAccess::DirectoryEntry::findFirst()+0x14 at SimpleFile.cpp:600 (libhdbbasis.so)

14: 0x00007fd5634d3d9b in FileAccess::DirectoryEntry::DirectoryEntry(char const*)+0x467 at SimpleFile.cpp:99 (libhdbbasis.so)

15: 0x00007fd5633aaec0 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x250 at TraceSegment.cpp:158 (libhdbbasis.so)

16: 0x00007fd563471f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

17: 0x00007fd563472dfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

--

1642[thr=1285]: MemoryCompactor is inactive

--

1647[thr=1287]: NetworkChannelCompletionThread is inactive

--

1648[thr=1288]: NetworkChannelCompletionThread is inactive

--

1661[thr=1291]: WakeupThread is inactive

--

2231[thr=1294]: WebDispatcher-Main-Thread is inactive

--

2236[thr=1295]: SignalQueue is inactive

--

2249[thr=1300]: WebDispatcherWorkerThread0000 at

1: 0x00007fd56220daa1 in sem_timedwait+0x60 (libpthread.so.0)

2: 0x00007fd56366745b in Synchronization::InterProcessMutex::waitEvent(void*, void*)+0x37 at InterProcessMutex.cpp:346 (libhdbbasis.so)

3: 0x00007fd563667723 in Synchronization::InterProcessMutex::timedLock(void*, unsigned long)+0x1a0 at InterProcessMutex.cpp:315 (libhdbbasis.so)

4: 0x00007fd563590abc in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLock(bool, unsigned long, unsigned long, bool&)+0x48 at GlobalMemoryHandler.cpp:6830 (libhdbbasis.so)

5: 0x00007fd563590d8c in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemoryHlp(MemoryManager::IpmmMasterScope&, unsigned long)+0x88 at GlobalMemoryHandler.cpp:6804 (libhdbbasis.so)

6: 0x00007fd563590eb8 in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemory(MemoryManager::IpmmMasterScope&, ltt::auto_object<Synchronization::LockedScope<Synchronization::NamedProcessMutexNoAlloc, true> >&, unsigned long, bool)+0xe4 at GlobalMemoryHandler.cpp:6762 (libhdbbasis.so)

7: 0x00007fd5635a015c in MemoryManager::GlobalMemoryHandler::provideMemoryAndReturnSize(bool, bool, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x698 at GlobalMemoryHandler.cpp:4603 (libhdbbasis.so)

8: 0x00007fd5635a0cd5 in MemoryManager::GlobalMemoryHandler::provideMemory(unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x41 at GlobalMemoryHandler.cpp:4387 (libhdbbasis.so)

9: 0x00007fd56353fa45 in MemoryManager::AllocatorConfigurationSPI::provideMemory(unsigned long, unsigned long, bool, char const*, unsigned long)+0x21 at AllocatorConfigurationImpl.cpp:2765 (libhdbbasis.so)

10: 0x00007fd56360c0d1 in MemoryManager::VirtualAllocTracer::reserveMemory(unsigned long&, bool, bool, void*, unsigned long)+0x290 at VirtualAllocTracer.cpp:187 (libhdbbasis.so)

11: 0x00007fd5635ade6c in MemoryManager::mmapOverride(void*, unsigned long, int, int, int, long, void*)+0x2b8 at MallocProxy.cpp:1660 (libhdbbasis.so)

12: 0x00007fd561ed22ec in _IO_file_doallocate_internal+0x78 (libc.so.6)

13: 0x00007fd561edf7ac in _IO_doallocbuf_internal+0x28 (libc.so.6)

14: 0x00007fd561eddd64 in _IO_new_file_underflow+0x1b0 (libc.so.6)

15: 0x00007fd561edf7ee in _IO_default_uflow_internal+0xa (libc.so.6)

16: 0x00007fd561ed3eca in _IO_getline_info_internal+0xa6 (libc.so.6)

17: 0x00007fd561edcd01 in __GI_fgets_unlocked+0x50 (libc.so.6)

18: 0x00007fd517f606cf in _nss_files_gethostbyname2_r+0x12b (libnss_files.so.2)

19: 0x00007fd561f3c910 in gaih_inet+0x1170 (libc.so.6)

20: 0x00007fd561f3edc0 in __GI_getaddrinfo+0x150 (libc.so.6)

21: 0x00007fd48cc37a1b in NiPGetHostByName(char const*, unsigned char, unsigned char, NI_NODEADDR*, unsigned int*, char*, unsigned int, _IO_FILE**)+0xa7 (hdbwebdispatcherslib.so)

22: 0x00007fd48cc5c5e8 in NIHIMPL_LINEAR::getNodeAddr(char const*, NI_NODEADDR*, unsigned int, int, _IO_FILE**)+0xc4 (hdbwebdispatcherslib.so)

23: 0x00007fd48cc63a42 in NiIGetNodeAddr(char const*, int, NI_NODEADDR*, unsigned int, _IO_FILE**)+0x1b0 (hdbwebdispatcherslib.so)

24: 0x00007fd48c95f635 in HttpModHandler(HTTP_SUB_HANDLER_DATA*, ICM_PLUGIN_PARAM*, HTTP_TASK, IctIHttpMsgAdm*, IctIHttpMsgAdm*, _ICM_BUFFER*, unsigned long, void**)+0x50d1 (hdbwebdispatcherslib.so)

25: 0x00007fd48c946db8 in HttpSrvHdlRequest(ICM_PLUGIN_PARAM*, PLUGIN_DATA*, _ICM_BUFFER*, unsigned char)+0xc4 (hdbwebdispatcherslib.so)

26: 0x00007fd48c948e4e in HttpPlugInHandleNetData+0xa4a (hdbwebdispatcherslib.so)

27: 0x00007fd48c9faae0 in IcmPlugInHandleNetData+0xb0 (hdbwebdispatcherslib.so)

28: 0x00007fd48ca1899d in IcmHandleNetRead(_ICM_REQUEST*, int)+0x199 (hdbwebdispatcherslib.so)

29: 0x00007fd48ca19e12 in IcmWorkerThread+0x940 (hdbwebdispatcherslib.so)

30: 0x00007fd563470098 in Execution::pthreadFunctionWrapper(void*)+0x94 at Thread.cpp:1239 (libhdbbasis.so)

31: 0x00007fd562207a51 in start_thread+0xd0 (libpthread.so.0)

32: 0x00007fd561f5493d in clone+0x69 (libc.so.6)

--

2250[thr=1301]: WebDispatcherWorkerThread0001 is inactive

--

2251[thr=1302]: WebDispatcherWorkerThread0002 at

1: 0x00007fd56220e2e4 in __lll_lock_wait+0x20 (libpthread.so.0)

2: 0x00007fd562209588 in _L_lock_854+0xb (libpthread.so.0)

3: 0x00007fd562209457 in __pthread_mutex_lock_internal+0x33 (libpthread.so.0)

4: 0x00007fd517f60606 in _nss_files_gethostbyname2_r+0x62 (libnss_files.so.2)

5: 0x00007fd561f3c910 in gaih_inet+0x1170 (libc.so.6)

6: 0x00007fd561f3edc0 in __GI_getaddrinfo+0x150 (libc.so.6)

7: 0x00007fd48cc37a1b in NiPGetHostByName(char const*, unsigned char, unsigned char, NI_NODEADDR*, unsigned int*, char*, unsigned int, _IO_FILE**)+0xa7 (hdbwebdispatcherslib.so)

8: 0x00007fd48cc5c5e8 in NIHIMPL_LINEAR::getNodeAddr(char const*, NI_NODEADDR*, unsigned int, int, _IO_FILE**)+0xc4 (hdbwebdispatcherslib.so)

9: 0x00007fd48cc63a42 in NiIGetNodeAddr(char const*, int, NI_NODEADDR*, unsigned int, _IO_FILE**)+0x1b0 (hdbwebdispatcherslib.so)

10: 0x00007fd48c95f635 in HttpModHandler(HTTP_SUB_HANDLER_DATA*, ICM_PLUGIN_PARAM*, HTTP_TASK, IctIHttpMsgAdm*, IctIHttpMsgAdm*, _ICM_BUFFER*, unsigned long, void**)+0x50d1 (hdbwebdispatcherslib.so)

11: 0x00007fd48c946db8 in HttpSrvHdlRequest(ICM_PLUGIN_PARAM*, PLUGIN_DATA*, _ICM_BUFFER*, unsigned char)+0xc4 (hdbwebdispatcherslib.so)

12: 0x00007fd48c948e4e in HttpPlugInHandleNetData+0xa4a (hdbwebdispatcherslib.so)

13: 0x00007fd48c9faae0 in IcmPlugInHandleNetData+0xb0 (hdbwebdispatcherslib.so)

14: 0x00007fd48ca1899d in IcmHandleNetRead(_ICM_REQUEST*, int)+0x199 (hdbwebdispatcherslib.so)

15: 0x00007fd48ca19e12 in IcmWorkerThread+0x940 (hdbwebdispatcherslib.so)

16: 0x00007fd563470098 in Execution::pthreadFunctionWrapper(void*)+0x94 at Thread.cpp:1239 (libhdbbasis.so)

17: 0x00007fd562207a51 in start_thread+0xd0 (libpthread.so.0)

18: 0x00007fd561f5493d in clone+0x69 (libc.so.6)

--

2252[thr=1303]: WebDispatcherWorkerThread0003 is inactive

--

2253[thr=1304]: WebDispatcherWorkerThread0004 is inactive

--

2254[thr=1305]: WebDispatcherWorkerThread0005 at

1: 0x00007fd56220e2e4 in __lll_lock_wait+0x20 (libpthread.so.0)

2: 0x00007fd562209588 in _L_lock_854+0xb (libpthread.so.0)

3: 0x00007fd562209457 in __pthread_mutex_lock_internal+0x33 (libpthread.so.0)

4: 0x00007fd517f60606 in _nss_files_gethostbyname2_r+0x62 (libnss_files.so.2)

5: 0x00007fd561f3c910 in gaih_inet+0x1170 (libc.so.6)

6: 0x00007fd561f3edc0 in __GI_getaddrinfo+0x150 (libc.so.6)

7: 0x00007fd48cc37a1b in NiPGetHostByName(char const*, unsigned char, unsigned char, NI_NODEADDR*, unsigned int*, char*, unsigned int, _IO_FILE**)+0xa7 (hdbwebdispatcherslib.so)

8: 0x00007fd48cc5c5e8 in NIHIMPL_LINEAR::getNodeAddr(char const*, NI_NODEADDR*, unsigned int, int, _IO_FILE**)+0xc4 (hdbwebdispatcherslib.so)

9: 0x00007fd48cc63a42 in NiIGetNodeAddr(char const*, int, NI_NODEADDR*, unsigned int, _IO_FILE**)+0x1b0 (hdbwebdispatcherslib.so)

10: 0x00007fd48c95f635 in HttpModHandler(HTTP_SUB_HANDLER_DATA*, ICM_PLUGIN_PARAM*, HTTP_TASK, IctIHttpMsgAdm*, IctIHttpMsgAdm*, _ICM_BUFFER*, unsigned long, void**)+0x50d1 (hdbwebdispatcherslib.so)

11: 0x00007fd48c946db8 in HttpSrvHdlRequest(ICM_PLUGIN_PARAM*, PLUGIN_DATA*, _ICM_BUFFER*, unsigned char)+0xc4 (hdbwebdispatcherslib.so)

12: 0x00007fd48c948e4e in HttpPlugInHandleNetData+0xa4a (hdbwebdispatcherslib.so)

13: 0x00007fd48c9faae0 in IcmPlugInHandleNetData+0xb0 (hdbwebdispatcherslib.so)

14: 0x00007fd48ca1899d in IcmHandleNetRead(_ICM_REQUEST*, int)+0x199 (hdbwebdispatcherslib.so)

15: 0x00007fd48ca19e12 in IcmWorkerThread+0x940 (hdbwebdispatcherslib.so)

16: 0x00007fd563470098 in Execution::pthreadFunctionWrapper(void*)+0x94 at Thread.cpp:1239 (libhdbbasis.so)

17: 0x00007fd562207a51 in start_thread+0xd0 (libpthread.so.0)

18: 0x00007fd561f5493d in clone+0x69 (libc.so.6)

--

2255[thr=1306]: WebDispatcherWorkerThread0006 at

1: 0x00007fd56220e2e4 in __lll_lock_wait+0x20 (libpthread.so.0)

2: 0x00007fd562209588 in _L_lock_854+0xb (libpthread.so.0)

3: 0x00007fd562209457 in __pthread_mutex_lock_internal+0x33 (libpthread.so.0)

4: 0x00007fd517f60606 in _nss_files_gethostbyname2_r+0x62 (libnss_files.so.2)

5: 0x00007fd561f3c910 in gaih_inet+0x1170 (libc.so.6)

6: 0x00007fd561f3edc0 in __GI_getaddrinfo+0x150 (libc.so.6)

7: 0x00007fd48cc37a1b in NiPGetHostByName(char const*, unsigned char, unsigned char, NI_NODEADDR*, unsigned int*, char*, unsigned int, _IO_FILE**)+0xa7 (hdbwebdispatcherslib.so)

8: 0x00007fd48cc5c5e8 in NIHIMPL_LINEAR::getNodeAddr(char const*, NI_NODEADDR*, unsigned int, int, _IO_FILE**)+0xc4 (hdbwebdispatcherslib.so)

9: 0x00007fd48cc63a42 in NiIGetNodeAddr(char const*, int, NI_NODEADDR*, unsigned int, _IO_FILE**)+0x1b0 (hdbwebdispatcherslib.so)

10: 0x00007fd48c95f635 in HttpModHandler(HTTP_SUB_HANDLER_DATA*, ICM_PLUGIN_PARAM*, HTTP_TASK, IctIHttpMsgAdm*, IctIHttpMsgAdm*, _ICM_BUFFER*, unsigned long, void**)+0x50d1 (hdbwebdispatcherslib.so)

11: 0x00007fd48c946db8 in HttpSrvHdlRequest(ICM_PLUGIN_PARAM*, PLUGIN_DATA*, _ICM_BUFFER*, unsigned char)+0xc4 (hdbwebdispatcherslib.so)

12: 0x00007fd48c948e4e in HttpPlugInHandleNetData+0xa4a (hdbwebdispatcherslib.so)

13: 0x00007fd48c9faae0 in IcmPlugInHandleNetData+0xb0 (hdbwebdispatcherslib.so)

14: 0x00007fd48ca1899d in IcmHandleNetRead(_ICM_REQUEST*, int)+0x199 (hdbwebdispatcherslib.so)

15: 0x00007fd48ca19e12 in IcmWorkerThread+0x940 (hdbwebdispatcherslib.so)

16: 0x00007fd563470098 in Execution::pthreadFunctionWrapper(void*)+0x94 at Thread.cpp:1239 (libhdbbasis.so)

17: 0x00007fd562207a51 in start_thread+0xd0 (libpthread.so.0)

18: 0x00007fd561f5493d in clone+0x69 (libc.so.6)

--

[OK]

--

[PROCESS_INFO]  Process Info: (2016-04-25 16:17:30 418 Local)

system pagesize: 4096

dumping process info, all memory sizes in system pages!

  PID      PPID        UID      VIRT        RES        SHR        TRS        DRS        LRS COMMAND

    1          0          0      4841        386        306        35          0        105 /sbin/init

    2          0          0          0          0          0          0          0          0 [kthreadd]

    3          2          0          0          0          0          0          0          0 [migration/0]

    4          2          0          0          0          0          0          0          0 [ksoftirqd/0]

    5          2          0          0          0          0          0          0          0 [migration/0]

    6          2          0          0          0          0          0          0          0 [watchdog/0]

    7          2          0          0          0          0          0          0          0 [migration/1]

    8          2          0          0          0          0          0          0          0 [migration/1]

    9          2          0          0          0          0          0          0          0 [ksoftirqd/1]

    10          2          0          0          0          0          0          0          0 [watchdog/1]

    11          2          0          0          0          0          0          0          0 [migration/2]

    12          2          0          0          0          0          0          0          0 [migration/2]

    13          2          0          0          0          0          0          0          0 [ksoftirqd/2]

    14          2          0          0          0          0          0          0          0 [watchdog/2]

    15          2          0          0          0          0          0          0          0 [migration/3]

    16          2          0          0          0          0          0          0          0 [migration/3]

    17          2          0          0          0          0          0          0          0 [ksoftirqd/3]

    18          2          0          0          0          0          0          0          0 [watchdog/3]

    19          2          0          0          0          0          0          0          0 [migration/4]

    20          2          0          0          0          0          0          0          0 [migration/4]

    21          2          0          0          0          0          0          0          0 [ksoftirqd/4]

    22          2          0          0          0          0          0          0          0 [watchdog/4]

    23          2          0          0          0          0          0          0          0 [migration/5]

    24          2          0          0          0          0          0          0          0 [migration/5]

    25          2          0          0          0          0          0          0          0 [ksoftirqd/5]

    26          2          0          0          0          0          0          0          0 [watchdog/5]

    27          2          0          0          0          0          0          0          0 [migration/6]

    28          2          0          0          0          0          0          0          0 [migration/6]

    29          2          0          0          0          0          0          0          0 [ksoftirqd/6]

    30          2          0          0          0          0          0          0          0 [watchdog/6]

    31          2          0          0          0          0          0          0          0 [migration/7]

    32          2          0          0          0          0          0          0          0 [migration/7]

    33          2          0          0          0          0          0          0          0 [ksoftirqd/7]

    34          2          0          0          0          0          0          0          0 [watchdog/7]

    35          2          0          0          0          0          0          0          0 [events/0]

    36          2          0          0          0          0          0          0          0 [events/1]

    37          2          0          0          0          0          0          0          0 [events/2]

    38          2          0          0          0          0          0          0          0 [events/3]

    39          2          0          0          0          0          0          0          0 [events/4]

    40          2          0          0          0          0          0          0          0 [events/5]

    41          2          0          0          0          0          0          0          0 [events/6]

    42          2          0          0          0          0          0          0          0 [events/7]

    43          2          0          0          0          0          0          0          0 [cgroup]

    44          2          0          0          0          0          0          0          0 [khelper]

    45          2          0          0          0          0          0          0          0 [netns]

    46          2          0          0          0          0          0          0          0 [async/mgr]

    47          2          0          0          0          0          0          0          0 [pm]

    48          2          0          0          0          0          0          0          0 [sync_supers]

    49          2          0          0          0          0          0          0          0 [bdi-default]

    50          2          0          0          0          0          0          0          0 [kintegrityd/0]

    51          2          0          0          0          0          0          0          0 [kintegrityd/1]

    52          2          0          0          0          0          0          0          0 [kintegrityd/2]

    53          2          0          0          0          0          0          0          0 [kintegrityd/3]

    54          2          0          0          0          0          0          0          0 [kintegrityd/4]

    55          2          0          0          0          0          0          0          0 [kintegrityd/5]

    56          2          0          0          0          0          0          0          0 [kintegrityd/6]

    57          2          0          0          0          0          0          0          0 [kintegrityd/7]

    58          2          0          0          0          0          0          0          0 [kblockd/0]

    59          2          0          0          0          0          0          0          0 [kblockd/1]

    60          2          0          0          0          0          0          0          0 [kblockd/2]

    61          2          0          0          0          0          0          0          0 [kblockd/3]

    62          2          0          0          0          0          0          0          0 [kblockd/4]

    63          2          0          0          0          0          0          0          0 [kblockd/5]

    64          2          0          0          0          0          0          0          0 [kblockd/6]

    65          2          0          0          0          0          0          0          0 [kblockd/7]

    66          2          0          0          0          0          0          0          0 [kacpid]

    67          2          0          0          0          0          0          0          0 [kacpi_notify]

    68          2          0          0          0          0          0          0          0 [kacpi_hotplug]

    69          2          0          0          0          0          0          0          0 [ata_aux]

    70          2          0          0          0          0          0          0          0 [ata_sff/0]

    71          2          0          0          0          0          0          0          0 [ata_sff/1]

    72          2          0          0          0          0          0          0          0 [ata_sff/2]

    73          2          0          0          0          0          0          0          0 [ata_sff/3]

    74          2          0          0          0          0          0          0          0 [ata_sff/4]

    75          2          0          0          0          0          0          0          0 [ata_sff/5]

    76          2          0          0          0          0          0          0          0 [ata_sff/6]

    77          2          0          0          0          0          0          0          0 [ata_sff/7]

    78          2          0          0          0          0          0          0          0 [ksuspend_usbd]

    79          2          0          0          0          0          0          0          0 [khubd]

    80          2          0          0          0          0          0          0          0 [kseriod]

    81          2          0          0          0          0          0          0          0 [md/0]

    82          2          0          0          0          0          0          0          0 [md/1]

    83          2          0          0          0          0          0          0          0 [md/2]

    84          2          0          0          0          0          0          0          0 [md/3]

    85          2          0          0          0          0          0          0          0 [md/4]

    86          2          0          0          0          0          0          0          0 [md/5]

    87          2          0          0          0          0          0          0          0 [md/6]

    88          2          0          0          0          0          0          0          0 [md/7]

    89          2          0          0          0          0          0          0          0 [md_misc/0]

    90          2          0          0          0          0          0          0          0 [md_misc/1]

    91          2          0          0          0          0          0          0          0 [md_misc/2]

    92          2          0          0          0          0          0          0          0 [md_misc/3]

    93          2          0          0          0          0          0          0          0 [md_misc/4]

    94          2          0          0          0          0          0          0          0 [md_misc/5]

    95          2          0          0          0          0          0          0          0 [md_misc/6]

    96          2          0          0          0          0          0          0          0 [md_misc/7]

    97          2          0          0          0          0          0          0          0 [linkwatch]

    98          2          0          0          0          0          0          0          0 [khungtaskd]

    99          2          0          0          0          0          0          0          0 [kswapd0]

  100          2          0          0          0          0          0          0          0 [ksmd]

  101          2          0          0          0          0          0          0          0 [aio/0]

  102          2          0          0          0          0          0          0          0 [aio/1]

  103          2          0          0          0          0          0          0          0 [aio/2]

  104          2          0          0          0          0          0          0          0 [aio/3]

  105          2          0          0          0          0          0          0          0 [aio/4]

  106          2          0          0          0          0          0          0          0 [aio/5]

  107          2          0          0          0          0          0          0          0 [aio/6]

  108          2          0          0          0          0          0          0          0 [aio/7]

  109          2          0          0          0          0          0          0          0 [crypto/0]

  110          2          0          0          0          0          0          0          0 [crypto/1]

  111          2          0          0          0          0          0          0          0 [crypto/2]

  112          2          0          0          0          0          0          0          0 [crypto/3]

  113          2          0          0          0          0          0          0          0 [crypto/4]

  114          2          0          0          0          0          0          0          0 [crypto/5]

  115          2          0          0          0          0          0          0          0 [crypto/6]

  116          2          0          0          0          0          0          0          0 [crypto/7]

  121          2          0          0          0          0          0          0          0 [kthrotld/0]

  122          2          0          0          0          0          0          0          0 [kthrotld/1]

  123          2          0          0          0          0          0          0          0 [kthrotld/2]

  124          2          0          0          0          0          0          0          0 [kthrotld/3]

  125          2          0          0          0          0          0          0          0 [kthrotld/4]

  126          2          0          0          0          0          0          0          0 [kthrotld/5]

  127          2          0          0          0          0          0          0          0 [kthrotld/6]

  128          2          0          0          0          0          0          0          0 [kthrotld/7]

  129          2          0          0          0          0          0          0          0 [pciehpd]

  131          2          0          0          0          0          0          0          0 [kpsmoused]

  132          2          0          0          0          0          0          0          0 [usbhid_resumer]

  162          2          0          0          0          0          0          0          0 [kstriped]

  243          2          0          0          0          0          0          0          0 [scsi_eh_0]

  246          2          0          0          0          0          0          0          0 [scsi_eh_1]

  253          2          0          0          0          0          0          0          0 [scsi_eh_2]

  254          2          0          0          0          0          0          0          0 [vmw_pvscsi_wq_2]

  438          2          0          0          0          0          0          0          0 [kdmflush]

  440          2          0          0          0          0          0          0          0 [kdmflush]

  457          2          0          0          0          0          0          0          0 [jbd2/dm-0-8]

  458          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

  550          1          0      2768        313        103        31          0        206 /sbin/udevd -d

  601          1      1000      6607        399        227        145          0        362 sapstart pf=/hana/shared/HED/profile/HED_HDB10_uyclsxxx.mdp.local

  610        601      1000    138893      71932      69195        185          0      25408 /usr/sap/HED/HDB10/uyclsxxx.mdp.local/trace/hdb.sapHED_HDB10 -d -nw -f /usr/sap/HED/HDB10/uyclsxxx.mdp.local/daemon.ini pf=/usr/sap/HED/SYS/profile/HED_HDB10_uyclsxxx.mdp.local

  628        610      1000    2567429    1953327    163654        544          0    1829207 hdbnameserver

  966        610      1000    706714    108534      44698        67          0    108220 hdbcompileserver

  968        610      1000    682613      94421      43260        753          0      80159 hdbpreprocessor

  989          2          0          0          0          0          0          0          0 [vmmemctl]

  995        610      1000  11385361  10674342    2854449        504          0    7896693 hdbindexserver -port 31043

  1038          2          0          0          0          0          0          0          0 [kdmflush]

  1080          2          0          0          0          0          0          0          0 [jbd2/sda1-8]

  1081          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

  1082          2          0          0          0          0          0          0          0 [jbd2/dm-4-8]

  1083          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

  1086          2          0          0          0          0          0          0          0 [xfs_mru_cache]

  1087          2          0          0          0          0          0          0          0 [xfslogd/0]

  1088          2          0          0          0          0          0          0          0 [xfslogd/1]

  1089          2          0          0          0          0          0          0          0 [xfslogd/2]

  1090          2          0          0          0          0          0          0          0 [xfslogd/3]

  1091          2          0          0          0          0          0          0          0 [xfslogd/4]

  1092          2          0          0          0          0          0          0          0 [xfslogd/5]

  1093          2          0          0          0          0          0          0          0 [xfslogd/6]

  1094          2          0          0          0          0          0          0          0 [xfslogd/7]

  1095          2          0          0          0          0          0          0          0 [xfsdatad/0]

  1096          2          0          0          0          0          0          0          0 [xfsdatad/1]

  1097          2          0          0          0          0          0          0          0 [xfsdatad/2]

  1098          2          0          0          0          0          0          0          0 [xfsdatad/3]

  1099          2          0          0          0          0          0          0          0 [xfsdatad/4]

  1100          2          0          0          0          0          0          0          0 [xfsdatad/5]

  1101          2          0          0          0          0          0          0          0 [xfsdatad/6]

  1102          2          0          0          0          0          0          0          0 [xfsdatad/7]

  1103          2          0          0          0          0          0          0          0 [xfsconvertd/0]

  1104          2          0          0          0          0          0          0          0 [xfsconvertd/1]

  1105          2          0          0          0          0          0          0          0 [xfsconvertd/2]

  1106          2          0          0          0          0          0          0          0 [xfsconvertd/3]

  1107          2          0          0          0          0          0          0          0 [xfsconvertd/4]

  1108          2          0          0          0          0          0          0          0 [xfsconvertd/5]

  1109          2          0          0          0          0          0          0          0 [xfsconvertd/6]

  1110          2          0          0          0          0          0          0          0 [xfsconvertd/7]

  1111          2          0          0          0          0          0          0          0 [xfsalloc/0]

  1112          2          0          0          0          0          0          0          0 [xfsalloc/1]

  1113          2          0          0          0          0          0          0          0 [xfsalloc/2]

  1114          2          0          0          0          0          0          0          0 [xfsalloc/3]

  1115          2          0          0          0          0          0          0          0 [xfsalloc/4]

  1116          2          0          0          0          0          0          0          0 [xfsalloc/5]

  1117          2          0          0          0          0          0          0          0 [xfsalloc/6]

  1118          2          0          0          0          0          0          0          0 [xfsalloc/7]

  1119          2          0          0          0          0          0          0          0 [xfseofblocks/0]

  1120          2          0          0          0          0          0          0          0 [xfseofblocks/1]

  1121          2          0          0          0          0          0          0          0 [xfseofblocks/2]

  1122          2          0          0          0          0          0          0          0 [xfseofblocks/3]

  1123          2          0          0          0          0          0          0          0 [xfseofblocks/4]

  1124          2          0          0          0          0          0          0          0 [xfseofblocks/5]

  1125          2          0          0          0          0          0          0          0 [xfseofblocks/6]

  1126          2          0          0          0          0          0          0          0 [xfseofblocks/7]

  1175          2          0          0          0          0          0          0          0 [kauditd]

  1272        610      1000    760563    106827      44605        630          0    155509 hdbwebdispatcher

  1378          1          0      23294        222        154        41          0      19022 auditd

  1394          1          0      62319        734        277        89          0      57003 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5

  1423          1          0      4596        197        118        10          0        108 irqbalance --pid=/var/run/irqbalance.pid

  1437          1          0      4744        227        165        13          0        76 rpcbind

  1455          1          0      5837        341        229        17          0        131 rpc.statd

  1481          1          0      7912        289        208        81          0      2634 dbus-daemon --system

  1511          1          0      1020        165        137          6          0        67 /usr/sbin/acpid

  1520          1          0      9502        945        702        83          0      2799 hald

  1521      1520          0      5100        301        245          4          0        75 hald-runner

  1553      1521          0      5630        275        235          6          0        75 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0

  1573      1521          0      4502        258        222          3          0        74 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket

  1799          1          0      16663        309        131        136          0        155 /usr/sbin/sshd

  1822          1          0      7684        529        375        173          0        198 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g

  1898          1          0      20324        861        635        36          0        150 /usr/libexec/postfix/master

  1907      1898          0      20387        862        640        72          0        177 qmgr -l -t fifo -u

  1908          1          0      44629        472        352          3          0        117 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages

  1919          1          0      44313      1523        458          1          0      1038 /usr/bin/python /usr/sbin/tuned -d -c /etc/tuned.conf

  1929          1          0      29323        345        191        14          0        355 crond

  1940          1          0      5385        123        77          5          0        120 /usr/sbin/atd

  1953          1          0      27085        164        129          3          0        72 /usr/bin/rhsmcertd

  1967          1          0      1016        137        120          3          0        67 /sbin/mingetty /dev/tty1

  1969          1          0      1016        137        120          3          0        67 /sbin/mingetty /dev/tty2

  1971          1          0      1016        137        120          3          0        67 /sbin/mingetty /dev/tty3

  1973          1          0      1016        137        120          3          0        67 /sbin/mingetty /dev/tty4

  1975          1          0      1016        138        120          3          0        67 /sbin/mingetty /dev/tty5

  1977          1          0      1016        138        120          3          0        67 /sbin/mingetty /dev/tty6

  2268          2          0          0          0          0          0          0          0 [flush-253:0]

  2406          2          0          0          0          0          0          0          0 [kdmflush]

  2417          2          0          0          0          0          0          0          0 [kdmflush]

  2715          2          0          0          0          0          0          0          0 [kdmflush]

  3017          2          0          0          0          0          0          0          0 [kdmflush]

  3085          2          0          0          0          0          0          0          0 [kdmflush]

  3560          1          0      30414        712        157          1          0        561 /usr/bin/python /usr/libexec/tuned/pmqos-static.py cpu_dma_latency=1

  3663          2          0          0          0          0          0          0          0 [flush-253:5]

  3728          2          0          0          0          0          0          0          0 [xfsbufd/dm-7]

  3729          2          0          0          0          0          0          0          0 [xfssyncd/dm-7]

  3730          2          0          0          0          0          0          0          0 [xfsaild/dm-7]

  3731          2          0          0          0          0          0          0          0 [xfsbufd/dm-3]

  3732          2          0          0          0          0          0          0          0 [xfssyncd/dm-3]

  3733          2          0          0          0          0          0          0          0 [xfsaild/dm-3]

  3734          2          0          0          0          0          0          0          0 [xfsbufd/dm-5]

  3735          2          0          0          0          0          0          0          0 [xfssyncd/dm-5]

  3736          2          0          0          0          0          0          0          0 [xfsaild/dm-5]

  3737          2          0          0          0          0          0          0          0 [xfsbufd/dm-6]

  3738          2          0          0          0          0          0          0          0 [xfssyncd/dm-6]

  3739          2          0          0          0          0          0          0          0 [xfsaild/dm-6]

  3740          2          0          0          0          0          0          0          0 [xfsbufd/dm-2]

  3741          2          0          0          0          0          0          0          0 [xfssyncd/dm-2]

  3742          2          0          0          0          0          0          0          0 [xfsaild/dm-2]

  3912          2          0          0          0          0          0          0          0 [flush-253:6]

  7008        610      1000  22229798  21431269    2940534        504          0  18659210 hdbindexserver -port 31040

  8965      1799          0      25099      1023        777        136          0        200 sshd: root@notty 

  8969      8965          0      14424        570        417        14          0        143 /usr/libexec/openssh/sftp-server

  9040      1799          0      25101      1109        852        136          0        202 sshd: root@pts/1 

  9044      9040          0      27085        471        365        212          0        106 -bash

  9062          2          0          0          0          0          0          0          0 [flush-253:3]

11057          1      1000    267694      15714      1669      2885          0    251696 /usr/sap/HED/HDB10/exe/sapstartsrv pf=/hana/shared/HED/profile/HED_HDB10_uyclsxxx.mdp.local -D -u hedadm

13614      9044          0      43868        667        512        28          0        178 sudo su - hedadm

13615      13614          0      36359        413        315          8          0        137 su - hedadm

13616      13615      1000      27114        478        369        212          0        135 -sh

23758        550          0      2767        279        70        31          0        205 /sbin/udevd -d

23759        550          0      3163        730        135        31          0        601 /sbin/udevd -d

23870          1          0      47395      1117        903        148          0      2864 /usr/sbin/vmtoolsd

24227      1898          0      20344        849        630        55          0        151 pickup -l -t fifo -u

29389          1          0      27512      2226        449        934          0      18777 /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile

29392          1        498    250193      16673      11687      3060          0    224935 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D

29422          1          0      6913      1178        552        743          0      1017 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile

[OK]

--

[OS_MEMORY]  Operating system information about memory: (2016-04-25 16:17:33 890 Local)

/proc/meminfo:

MemTotal:      154912776 kB

MemFree:        1766132 kB

Buffers:          154964 kB

Cached:        37543100 kB

SwapCached:            0 kB

Active:        120618356 kB

Inactive:      30313196 kB

Active(anon):  113299852 kB

Inactive(anon): 22791284 kB

Active(file):    7318504 kB

Inactive(file):  7521912 kB

Unevictable:          0 kB

Mlocked:              0 kB

SwapTotal:      6291448 kB

SwapFree:        6291448 kB

Dirty:              7364 kB

Writeback:            0 kB

AnonPages:      113233420 kB

Mapped:        23267624 kB

Shmem:          22857644 kB

Slab:            737504 kB

SReclaimable:    615052 kB

SUnreclaim:      122452 kB

KernelStack:        5416 kB

PageTables:      394616 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:    83747836 kB

Committed_AS:  137030008 kB

VmallocTotal:  34359738367 kB

VmallocUsed:      542572 kB

VmallocChunk:  34359179772 kB

HardwareCorrupted:    0 kB

AnonHugePages:        0 kB

HugePages_Total:      0

HugePag

/proc/vmstat:

nr_free_pages 441533

nr_inactive_anon 5697821

nr_active_anon 28324963

nr_inactive_file 1880478

nr_active_file 1829626

nr_unevictable 0

nr_mlock 0

nr_anon_pages 28308355

nr_mapped 5816906

nr_file_pages 9424516

nr_dirty 1841

nr_writeback 0

nr_slab_reclaimable 153763

nr_slab_unreclaimable 30613

nr_page_table_pages 98654

nr_kernel_stack 677

nr_unstable 0

nr_bounce 0

nr_vmscan_write 433

nr_writeback_temp 0

nr_isolated_anon 0

nr_isolated_file 0

nr_shmem 5714411

numa_hit 2124894219

numa_miss 0

numa_foreign 0

numa_interleave 114388

numa_local 2124894219

numa_other 0

nr_anon_transparent_hugepages 0

pgpgin 1565579798

pgpgout 2795670662

pswpin 433

pswpout 433

pgalloc_dma 2

pgalloc_dma32 155436

pgalloc_normal 2130232508

pgalloc_movable 0

pgfree 2130831433

pgactivate 9298832

pgdeactivate 46976

pgfault 2180575674

pgmajfault 1254

pgrefill_dma 0

pgrefill_dma32 46976

pgrefill_normal 0

pgrefill_movable 0

pgsteal_dma 0

pgsteal_dma32 433

pgsteal_normal 2869225

pgsteal_movable 0

pgscan_kswapd_dma 0

pgscan_kswapd_dma32 2720

pgsca

/proc/sysvipc/shm:

      key      shmid perms                  size  cpid  lpid nattch  uid  gid  cuid  cgid      atime      dtime      ctime                  rss                  swap

        0  51118080  740                  1024  601  610      1  1000    79  1000    79 1460640107 1460640107 1460640106                    0                    0

    11001  51150849  600            268435456  610  7074      7  1000    79  1000    79 1460643197 1460643197 1460640107            268435456                    0

1446068590  51183618  660                  117  628  7008      6  1000  500  1000    79 1460643180 1460640195 1460640109                  4096                    0

351291274  51216387  660              1060911  628  7008    24  1000  500  1000    79 1460643180 1460640195 1460640109              1060864                    0

785145360  51249156  660                  108  628  7008      6  1000  500  1000    79 1460643181 1460640195 1460640109                  4096                    0

/proc/sys/kernel:

/proc/sys/vm/overcommit_memory:

0

/proc/sys/vm/panic_on_oom:

0

/proc/sys/vm/oom_kill_allocating_task:

0

/proc/sys/vm/compact_memory:

Could not open!

/proc/sys/vm/extfrag_threshold:

500

/proc/sys/vm/oom_dump_tasks:

1

/proc/sys/vm/would_have_oomkilled:

0

/proc/sys/vm/overcommit_ratio:

50

/proc/sys/vm/page-cluster:

3

/proc/sys/vm/dirty_background_ratio:

10

/proc/sys/vm/dirty_background_bytes:

0

/proc/sys/vm/dirty_ratio:

40

/proc/sys/vm/dirty_bytes:

0

/proc/sys/vm/dirty_writeback_centisecs:

500

/proc/sys/vm/dirty_expire_centisecs:

3000

/proc/sys/vm/nr_pdflush_threads:

0

/proc/sys/vm/swappiness:

60

/proc/sys/vm/nr_hugepages:

0

/proc/sys/vm/nr_hugepages_mempolicy:

0

/proc/sys/vm/hugetlb_shm_group:

0

/proc/sys/vm/hugepages_treat_as_movable:

0

/proc/sys/vm/nr_overcommit_hugepages:

0

/proc/sys/vm/lowmem_reserve_ratio:

256 256 32

/proc/sys/vm/drop_caches:

0

/proc/sys/vm/min_free_kbytes:

49818

/proc/sys/vm/extra_free_kbytes:

0

/proc/sys/vm/unmap_area_factor:

0

/proc/sys/vm/percpu_pagelist_fraction:

0

/proc/sys/vm/max_map_count:

15800000

/proc/sys/vm/laptop_mode:

0

/proc/sys/vm/block_dump:

0

/proc/sys/vm/vfs_cache_pressure:

100

/proc/sys/vm/legacy_va_layout:

0

/proc/sys/vm/zone_reclaim_mode:

0

/proc/sys/vm/min_unmapped_ratio:

1

/proc/sys/vm/min_slab_ratio:

5

/proc/sys/vm/stat_interval:

1

/proc/sys/vm/mmap_min_addr:

4096

/proc/sys/vm/numa_zonelist_order:

default

/proc/sys/vm/scan_unevictable_pages:

0

/proc/sys/vm/memory_failure_early_kill:

1

/proc/sys/vm/memory_failure_recovery:

1

[OK]

--

[THREADS]  Running threads: (2016-04-25 16:17:33 890 Local)

3082495[thr=3170]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2256[thr=1307]: WebDispatcherWorkerThread0007, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2257[thr=1308]: WebDispatcherWorkerThread0008, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2258[thr=1309]: WebDispatcherWorkerThread0009, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2259[thr=1310]: WebDispatcherWatchdogThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f54f33 in __epoll_wait_nocancel+0x2a (libc.so.6)

2285[thr=1311]: JobexWatchdog, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobexWatchdog", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2287[thr=1312]: JobexMainDispatcher, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2289[thr=1313]: JobWrkDiag, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobexDiag", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2291[thr=1314]: JobWrk0002, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2293[thr=1315]: JobWrk0003, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2306[thr=1318]: SelfWatchDog, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "SelfWatchDog", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2313[thr=1319]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2351[thr=1324]: JobWrk0004, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2806[thr=1718]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

212384[thr=22075]: JobWrk0005, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

470852[thr=11482]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

1430006[thr=16810]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

5[thr=0]: <SYSTEM>, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f4b113 in __libc_poll+0x53 (libc.so.6)

638[thr=1286]: TimerThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

667[thr=1289]: TraceSegmentCompressor, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: 1289, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "Search for trace files to compress", detail: "", command: "" at 0x00007fd5633e8c29 in Execution::ContextFunctions::printProgramCounter(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&)+0x19 at ContextFunctions.cpp:887 (libhdbbasis.so)

1642[thr=1285]: MemoryCompactor, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220daa1 in sem_timedwait+0x61 (libpthread.so.0)

1647[thr=1287]: NetworkChannelCompletionThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f54f33 in __epoll_wait_nocancel+0x2a (libc.so.6)

1648[thr=1288]: NetworkChannelCompletionThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f54f33 in __epoll_wait_nocancel+0x2a (libc.so.6)

1661[thr=1291]: WakeupThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220e7dd in __read_nocancel+0x24 (libpthread.so.0)

2231[thr=1294]: WebDispatcher-Main-Thread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd561f54f33 in __epoll_wait_nocancel+0x2a (libc.so.6)

2236[thr=1295]: SignalQueue, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "SignalQueue", method: "", detail: "", command: "" at 0x00007fd561f51199 in syscall+0x19 (libc.so.6)

2249[thr=1300]: WebDispatcherWorkerThread0000, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220daa1 in sem_timedwait+0x61 (libpthread.so.0)

2250[thr=1301]: WebDispatcherWorkerThread0001, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2251[thr=1302]: WebDispatcherWorkerThread0002, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220e2e4 in __lll_lock_wait+0x24 (libpthread.so.0)

2252[thr=1303]: WebDispatcherWorkerThread0003, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2253[thr=1304]: WebDispatcherWorkerThread0004, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220b63c in __pthread_cond_wait+0xcc (libpthread.so.0)

2254[thr=1305]: WebDispatcherWorkerThread0005, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220e2e4 in __lll_lock_wait+0x24 (libpthread.so.0)

2255[thr=1306]: WebDispatcherWorkerThread0006, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007fd56220e2e4 in __lll_lock_wait+0x24 (libpthread.so.0)

[OK]

--

[MEMORY_OOM]  Information about current out of memory situation: (2016-04-25 16:17:34 048 Local)

OUT OF MEMORY occurred.

Host: uyclsxxx.mdp.local

Executable: hdbwebdispatcher

PID: 1272

Failed to allocate 32816 byte.

Current callstack:

1: 0x00007fd5635f488c in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, unsigned long, bool)+0xb8 at PoolAllocator.cpp:777 (libhdbbasis.so)

2: 0x00007fd5635f55c0 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x250 at PoolAllocator.cpp:1449 (libhdbbasis.so)

3: 0x00007fd561f14651 in __alloc_dir+0x60 (libc.so.6)

4: 0x00007fd5636c02b4 in System::UX::opendir(char const*)+0x20 at SystemCallsUNIX.cpp:75 (libhdbbasis.so)

5: 0x00007fd5634cdde8 in FileAccess::DirectoryEntry::findFirst()+0x14 at SimpleFile.cpp:600 (libhdbbasis.so)

6: 0x00007fd5634d3d9b in FileAccess::DirectoryEntry::DirectoryEntry(char const*)+0x467 at SimpleFile.cpp:99 (libhdbbasis.so)

7: 0x00007fd5633aaec0 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x250 at TraceSegment.cpp:158 (libhdbbasis.so)

8: 0x00007fd563471f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

9: 0x00007fd563472dfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

Memory consumption information:

[No information available.]

Out of memory reasons:

!:! <1> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <2> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <3> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <4> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <5> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <6> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

At most one OOM dump is written in approximately 86400s.

Raw memory information (1):

Allocated=1359212544 (accounted=1359212544, unaccounted=0)

Reserved size for libraries (data segments only)=778547200 (data segments size (libraries)=778547200)

Raw Pool Allocator size=464121856 (including huge alignment raw size=0, V=0)

Allocate Only size=71471104(limited=16777216, unlimited=54693888 [including emergency allocators], unreserved Allocate Only size=0)Raw Limited Allocate Only Allocator size=16777216

Stack size=24858624

Free list size=0

Memory failure free list size=0

Big StackReservationScope reservations=0 (limited=0, unlimited=0)

VirtualAlloc/MMap = 19595264

JumpTable = 618496

StatAllocOwnBlocks =524288

OOMReservationAlloc (not part of 'allocated'!) = 0

(System memory: 13436020899840 - 13435439710208 = 581189632, + data sagements=1359736832)

Looks fine!

Compaction tasks

Last compaction: 538. Reason: IPMM (other process). Request ID=182737.

ReserveRequest=1.01mb (1060864b), ShrinkRequest=12.82mb (13447168b)

Compaction tasks for compaction 538:

  Limits:  TempLimit=1.26gb (1359212544b), ProcessLimit=138.79gb (149034151936b).

  Alloc status: Used=502.62mb (527036910b), Allocated=1.26gb (1359212544b).

  Fragmentation: 62%.

  Freelist: Size=0b.

* Forced garbage collection (fragmentation too big).

  Alloc status: Used=501.60mb (525976046b), Allocated=1.26gb (1360273408b).

  Fragmentation: 62%.

  Freelist: Size=0b.

  Compactor status before compaction: Size=0b, Freeable=0b.

* Weak garbage collection.

* Forced garbage collection..

  Alloc status: Used=501.60mb (525976046b), Allocated=1.26gb (1360273408b).

  Fragmentation: 62%.

  Freelist: Size=0b.

Result: Reserved=0b.

Called with parameter 'forced garbage collection'.

Called with parameter 'forced' (wait for lock).

Inter Process Memory Management

IPMM shared memory segment: SAP_IPMM_0028_HED

SId=        Process secondary identifier

AB=        Allocated bytes by allocators for heap and stacks (inclusive allocator cache)

UA=        Unaccounted memory (by unlimited allocators for exceptions etc.), maybe overcommit

U=          Used bytes for heap and stacks (without allocator cache)

FSL=        Free segment list size

SMAS/SMDS=  System memory traffic information (accumulation of allocations/deallocations)

PAL=        Process allocation limit

TPAL=      Temp process allocation limit

Blocked=    Memory (TPAL!) blocked/reserved for this process

W=          Weight of process

[0] PID=628 (state=S), SId=300674672, compactors active, alive, process name: hdbnameserver

    AB=7.72gb (8292470784b), UA=0b, U=5.89gb (6334672021b), FSL=0b, SMAS=7.90tb (8688838172672b), SMDS=7.89tb (8681396535296b), PAL=138.79gb (149034151936b), TPAL=7.72gb (8292470784b), W=100

    8 threads are waiting for IPMM lock

[1] PID=966 (state=S), SId=300674673, compactors active, alive, process name: hdbcompileserve

    AB=1.06gb (1143779328b), UA=0b, U=309.78mb (324837617b), FSL=0b, SMAS=27.95gb (30022529024b), SMDS=27.59gb (29628530688b), PAL=138.79gb (149034151936b), TPAL=2.20gb (2363437056b), Blocked=2.20gb (2363805696b), W=100

[2] PID=968 (state=S), SId=300674674, compactors active, alive, process name: hdbpreprocessor

    AB=988.55mb (1036574720b), UA=0b, U=217.32mb (227880645b), FSL=0b, SMAS=9.48tb (10425337434112b), SMDS=9.48tb (10425058766848b), PAL=138.79gb (149034151936b), TPAL=988.55mb (1036574720b), W=100

    2 threads are waiting for IPMM lock

[3] PID=995 (state=S), SId=300674675, compactors active, alive, process name: hdbindexserver

    AB=30.86gb (33144467456b), UA=0b, U=27.18gb (29185185267b), FSL=0b, SMAS=6.61tb (7272025485312b), SMDS=6.58tb (7239731965952b), PAL=138.79gb (149034151936b), TPAL=30.86gb (33144467456b), W=100

    6 threads are waiting for IPMM lock

[4] PID=1272 (state=R), SId=300674676, compactors active, alive, process name: hdbwebdispatche

    AB=1.26gb (1359212544b), UA=0b, U=502.62mb (527036910b), FSL=0b, SMAS=12.21tb (13436020899840b), SMDS=12.21tb (13435439710208b), PAL=138.79gb (149034151936b), TPAL=1.26gb (1359212544b), W=100

    2 threads are waiting for IPMM lock

[5] PID=7008 (state=S), SId=300674677, compactors active, alive, process name: hdbindexserver

    AB=72.17gb (77496229888b), UA=0b, U=67.06gb (72005726185b), FSL=0b, SMAS=9.83tb (10809720500224b), SMDS=9.76tb (10733343178752b), PAL=138.79gb (149034151936b), TPAL=72.17gb (77495169024b), W=100

    !! Requesting shrink...

    4 threads are waiting for IPMM lock

GLOBAL_MAX_ALLOCATION_LIMIT=138.79gb (149034151936b)

SHARED_MEMORY=21.69gb (23298207744b)

OOM_RESERVATION_ALLOCATORS=288.54mb (302563328b)

CODE_SIZE=1.62gb (1742049280b)

Sum TPAL (including shm, code size and IPMM reservation and IPMM OOM reservation)=138.79gb (149034151936b) (cached sum=138.79gb (149034151936b)), IPMM reservation=0b, emergency reservation=128mb (134217728b)

#processes=6, weight sum=600, cached weight sum=600, #checks for cached allocation limit sum=9668962, #terminated processes since last reset=0

Early exits are currently allowed. Dumping of IPMM flight recorder is currently enabled.

67 early exits since creation of IPMM, reclaim-memory-counter=1545607. Reclaim memory 1545605 failed, only 266833920b were free.

Last reclaim memory started to wait for IPMM lock since 2016-04-25 16:17:32.559.

Last reclaim memory got IPMM lock at 2016-04-25 16:17:33.869.

IPMM flight recorder is not verbose.

User ids respected in shm calculation: [1000]

Parallel memory reclaim tasks are activated.

This is an opt build.

Peak used memory: 125.63gb (134897761776b) with corresponding values (Heap=102.31gb (109857504752b), Shared=21.69gb (23298207744b), Code=1.62gb (1742049280b)).

Peak heap memory: 102.31gb (109857504752b). Peak shared memory: 21.69gb (23298207744b). Peak code size: 1.62gb (1742049280b)

IPMM flight recorder:

T=2016-04-25 16:16:09.947 (1461611769947254) C=1301: TPAL=1359212544, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.947 (1461611769947262) C=1301: Process 1272 incremented IPMM reserved bytes by 13598720 (->13598720) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.947 (1461611769947263) C=1301: Process 1272 uses IPMM reserved bytes (13598720) to increase reservation (-> 4096) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.947 (1461611769947266) C=1301: Process 1272 increments ProvideMemoryCounter: 1543453 -> 1543454

T=2016-04-25 16:16:09.947 (1461611769947286) C=1718: Process 1272 increments ProvideMemoryCounter: 1543454 -> 1543455

T=2016-04-25 16:16:09.947 (1461611769947290) C=1718: ==== Process 1272 must provide memory. Bytes to reserve=4096, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.947 (1461611769947291) C=1718: TPAL=1372811264, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.947 (1461611769947291) C=1718: Process 1272 reserves all memory until TPAL: 4096 (->4096), PM-inx=0

T=2016-04-25 16:16:09.947 (1461611769947294) C=1718: Process 1272 increments ProvideMemoryCounter: 1543455 -> 1543456

T=2016-04-25 16:16:09.947 (1461611769947303) C=1318: Process 1272 increments ProvideMemoryCounter: 1543456 -> 1543457

T=2016-04-25 16:16:09.947 (1461611769947305) C=1318: ==== Process 1272 must provide memory. Bytes to reserve=4096, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.947 (1461611769947305) C=1318: TPAL=1372811264, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.947 (1461611769947305) C=1318: Process 1272 reserves all memory until TPAL: 4096 (->4096), PM-inx=0

T=2016-04-25 16:16:09.947 (1461611769947307) C=1318: Process 1272 increments ProvideMemoryCounter: 1543457 -> 1543458

T=2016-04-25 16:16:09.947 (1461611769947853) C=1718: Process 1272 increments ProvideMemoryCounter: 1543458 -> 1543459

T=2016-04-25 16:16:09.947 (1461611769947856) C=1718: ==== Process 1272 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.947 (1461611769947856) C=1718: TPAL=1372811264, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.947 (1461611769947857) C=1718: Process 1272 reserves all memory until TPAL: 13598720 (->13598720), PM-inx=0

T=2016-04-25 16:16:09.947 (1461611769947861) C=1718: Process 1272 incremented IPMM reserved bytes by 84725760 (->84725760) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.947 (1461611769947862) C=1718: Process 1272 uses IPMM reserved bytes (84725760) to increase reservation (-> 98324480) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.951 (1461611769951042) C=1718: Process 1272 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:09.951 (1461611769951182) C=1718: Process 1272 incremented IPMM reserved bytes by 262144 (->262144) - reason: reduce TPAL of process 628.

T=2016-04-25 16:16:09.951 (1461611769951184) C=1718: Process 1272 incremented IPMM reserved bytes by 268173312 (->268435456) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.951 (1461611769951184) C=1718: Process 1272 uses IPMM reserved bytes (268435456) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.951 (1461611769951187) C=1718: Process 1272 increments ProvideMemoryCounter: 1543459 -> 1543460

T=2016-04-25 16:16:09.974 (1461611769974203) C=3900: Process 628 increments ProvideMemoryCounter: 1543460 -> 1543461

T=2016-04-25 16:16:09.974 (1461611769974209) C=3900: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.974 (1461611769974210) C=3900: TPAL=8296775680, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.974 (1461611769974217) C=3900: Process 628 incremented IPMM reserved bytes by 262144 (->262144) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.974 (1461611769974217) C=3900: Process 628 incremented IPMM reserved bytes by 98324480 (->98586624) - reason: reduce TPAL of process 1272.

T=2016-04-25 16:16:09.974 (1461611769974218) C=3900: Process 628 uses IPMM reserved bytes (98586624) to increase reservation (-> 98586624) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.978 (1461611769978083) C=3900: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:09.978 (1461611769978084) C=3900: Process 628 reserves all memory until TPAL: 262144 (->98848768), PM-inx=0

T=2016-04-25 16:16:09.978 (1461611769978353) C=3900: Process 628 incremented IPMM reserved bytes by 268435456 (->268435456) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.978 (1461611769978355) C=3900: Process 628 uses IPMM reserved bytes (268435456) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.978 (1461611769978360) C=3900: Process 628 increments ProvideMemoryCounter: 1543461 -> 1543462

T=2016-04-25 16:16:09.978 (1461611769978401) C=1027: Process 628 increments ProvideMemoryCounter: 1543462 -> 1543463

T=2016-04-25 16:16:09.978 (1461611769978405) C=1027: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.978 (1461611769978405) C=1027: TPAL=8663797760, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.978 (1461611769978406) C=1027: Process 628 reserves all memory until TPAL: 98848768 (->98848768), PM-inx=0

T=2016-04-25 16:16:09.978 (1461611769978409) C=1027: Process 628 incremented IPMM reserved bytes by 256225280 (->256225280) - reason: reduce TPAL of process 1272.

T=2016-04-25 16:16:09.978 (1461611769978410) C=1027: Process 628 uses IPMM reserved bytes (256225280) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.978 (1461611769978412) C=1027: Process 628 increments ProvideMemoryCounter: 1543463 -> 1543464

T=2016-04-25 16:16:09.978 (1461611769978434) C=1031: Process 628 increments ProvideMemoryCounter: 1543464 -> 1543465

T=2016-04-25 16:16:09.978 (1461611769978438) C=1031: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.978 (1461611769978438) C=1031: TPAL=8920023040, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.978 (1461611769978439) C=1031: Process 628 reserves all memory until TPAL: 86638592 (->86638592), PM-inx=0

T=2016-04-25 16:16:09.978 (1461611769978443) C=1031: Process 628 incremented IPMM reserved bytes by 12210176 (->12210176) - reason: reduce TPAL of process 1272.

T=2016-04-25 16:16:09.978 (1461611769978444) C=1031: Process 628 uses IPMM reserved bytes (12210176) to increase reservation (-> 98848768) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.983 (1461611769983181) C=1031: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:09.983 (1461611769983181) C=1031: Process 628 reserves all memory until TPAL: 169586688 (->268435456), PM-inx=0

T=2016-04-25 16:16:09.983 (1461611769983185) C=1031: Process 628 increments ProvideMemoryCounter: 1543465 -> 1543466

T=2016-04-25 16:16:09.987 (1461611769987360) C=1028: Process 628 increments ProvideMemoryCounter: 1543466 -> 1543467

T=2016-04-25 16:16:09.987 (1461611769987364) C=1028: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.987 (1461611769987364) C=1028: TPAL=8932233216, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.987 (1461611769987365) C=1028: Process 628 reserves all memory until TPAL: 98848768 (->98848768), PM-inx=0

T=2016-04-25 16:16:09.987 (1461611769987370) C=1028: Process 628 incremented IPMM reserved bytes by 405504 (->405504) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:09.987 (1461611769987371) C=1028: Process 628 uses IPMM reserved bytes (405504) to increase reservation (-> 99254272) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:09.989 (1461611769989985) C=1028: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:09.989 (1461611769989986) C=1028: Process 628 reserves all memory until TPAL: 169181184 (->268435456), PM-inx=0

T=2016-04-25 16:16:09.989 (1461611769989989) C=1028: Process 628 increments ProvideMemoryCounter: 1543467 -> 1543468

T=2016-04-25 16:16:09.990 (1461611769990787) C=645: Process 628 increments ProvideMemoryCounter: 1543468 -> 1543469

T=2016-04-25 16:16:09.990 (1461611769990791) C=645: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.990 (1461611769990791) C=645: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.990 (1461611769990792) C=645: Process 628 reserves all memory until TPAL: 268435456 (->268435456), PM-inx=0

T=2016-04-25 16:16:09.990 (1461611769990794) C=645: Process 628 increments ProvideMemoryCounter: 1543469 -> 1543470

T=2016-04-25 16:16:09.991 (1461611769991493) C=3903: Process 628 increments ProvideMemoryCounter: 1543470 -> 1543471

T=2016-04-25 16:16:09.991 (1461611769991496) C=3903: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.991 (1461611769991496) C=3903: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.991 (1461611769991497) C=3903: Process 628 reserves all memory until TPAL: 268435456 (->268435456), PM-inx=0

T=2016-04-25 16:16:09.991 (1461611769991499) C=3903: Process 628 increments ProvideMemoryCounter: 1543471 -> 1543472

T=2016-04-25 16:16:09.997 (1461611769997066) C=3894: Process 628 increments ProvideMemoryCounter: 1543472 -> 1543473

T=2016-04-25 16:16:09.997 (1461611769997072) C=3894: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:09.997 (1461611769997072) C=3894: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:09.997 (1461611769997073) C=3894: Process 628 reserves all memory until TPAL: 99254272 (->99254272), PM-inx=0

T=2016-04-25 16:16:10.000 (1461611770000225) C=3894: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.000 (1461611770000226) C=3894: Process 628 reserves all memory until TPAL: 1183744 (->100438016), PM-inx=0

T=2016-04-25 16:16:10.001 (1461611770001839) C=3894: Process 628 requested self compaction, to reserve=167997440, to shrink=188444672: Reserved 167997440!

T=2016-04-25 16:16:10.001 (1461611770001840) C=3894: Process 628 incremented IPMM reserved bytes by 534249472 (->534249472) - reason: compact own process (shrink request: 188444672).

T=2016-04-25 16:16:10.001 (1461611770001845) C=3894: Process 628 uses IPMM reserved bytes (534249472) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.001 (1461611770001850) C=3894: Process 628 increments ProvideMemoryCounter: 1543473 -> 1543474

T=2016-04-25 16:16:10.002 (1461611770002368) C=1030: Process 628 increments ProvideMemoryCounter: 1543474 -> 1543475

T=2016-04-25 16:16:10.002 (1461611770002373) C=1030: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.002 (1461611770002373) C=1030: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.002 (1461611770002373) C=1030: Process 628 reserves all memory until TPAL: 268435456 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.002 (1461611770002376) C=1030: Process 628 increments ProvideMemoryCounter: 1543475 -> 1543476

T=2016-04-25 16:16:10.002 (1461611770002403) C=1028: Process 628 increments ProvideMemoryCounter: 1543476 -> 1543477

T=2016-04-25 16:16:10.002 (1461611770002407) C=1028: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.002 (1461611770002407) C=1028: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.002 (1461611770002408) C=1028: Process 628 reserves all memory until TPAL: 97816576 (->97816576), PM-inx=0

T=2016-04-25 16:16:10.007 (1461611770007133) C=1028: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.007 (1461611770007133) C=1028: Process 628 reserves all memory until TPAL: 170618880 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.007 (1461611770007136) C=1028: Process 628 increments ProvideMemoryCounter: 1543477 -> 1543478

T=2016-04-25 16:16:10.008 (1461611770008200) C=1031: Process 628 increments ProvideMemoryCounter: 1543478 -> 1543479

T=2016-04-25 16:16:10.008 (1461611770008203) C=1031: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.008 (1461611770008203) C=1031: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.008 (1461611770008204) C=1031: Process 628 reserves all memory until TPAL: 96768000 (->96768000), PM-inx=0

T=2016-04-25 16:16:10.011 (1461611770011070) C=1031: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.011 (1461611770011071) C=1031: Process 628 reserves all memory until TPAL: 1048576 (->97816576), PM-inx=0

T=2016-04-25 16:16:10.015 (1461611770015126) C=1031: Process 628 requested self compaction, to reserve=170618880, to shrink=188628992: Reserved 170618880!

T=2016-04-25 16:16:10.015 (1461611770015128) C=1031: Process 628 incremented IPMM reserved bytes by 535240704 (->535240704) - reason: compact own process (shrink request: 188628992).

T=2016-04-25 16:16:10.015 (1461611770015132) C=1031: Process 628 uses IPMM reserved bytes (535240704) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.015 (1461611770015136) C=1031: Process 628 increments ProvideMemoryCounter: 1543479 -> 1543480

T=2016-04-25 16:16:10.016 (1461611770016659) C=1036: Process 995 increments ProvideMemoryCounter: 1543480 -> 1543481

T=2016-04-25 16:16:10.016 (1461611770016664) C=1036: ==== Process 995 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.016 (1461611770016664) C=1036: TPAL=33150459904, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.016 (1461611770016669) C=1036: Process 995 incremented IPMM reserved bytes by 364621824 (->364621824) - reason: reduce TPAL of process 628.

T=2016-04-25 16:16:10.016 (1461611770016670) C=1036: Process 995 uses IPMM reserved bytes (364621824) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.016 (1461611770016672) C=1036: Process 995 increments ProvideMemoryCounter: 1543481 -> 1543482

T=2016-04-25 16:16:10.019 (1461611770019059) C=3894: Process 628 increments ProvideMemoryCounter: 1543482 -> 1543483

T=2016-04-25 16:16:10.019 (1461611770019064) C=3894: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.019 (1461611770019064) C=3894: TPAL=8568016896, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.019 (1461611770019069) C=3894: Process 628 incremented IPMM reserved bytes by 96186368 (->96186368) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:10.019 (1461611770019072) C=3894: Process 628 uses IPMM reserved bytes (96186368) to increase reservation (-> 96186368) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.025 (1461611770025377) C=3894: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.025 (1461611770025378) C=3894: Process 628 reserves all memory until TPAL: 1630208 (->97816576), PM-inx=0

T=2016-04-25 16:16:10.025 (1461611770025629) C=3894: Process 628 incremented IPMM reserved bytes by 268435456 (->268435456) - reason: reduce TPAL of process 995.

T=2016-04-25 16:16:10.025 (1461611770025630) C=3894: Process 628 uses IPMM reserved bytes (268435456) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.025 (1461611770025636) C=3894: Process 628 increments ProvideMemoryCounter: 1543483 -> 1543484

T=2016-04-25 16:16:10.025 (1461611770025666) C=3901: Process 628 increments ProvideMemoryCounter: 1543484 -> 1543485

T=2016-04-25 16:16:10.025 (1461611770025670) C=3901: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.025 (1461611770025670) C=3901: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.025 (1461611770025671) C=3901: Process 628 reserves all memory until TPAL: 97816576 (->97816576), PM-inx=0

T=2016-04-25 16:16:10.032 (1461611770032120) C=3901: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.032 (1461611770032122) C=3901: Process 628 reserves all memory until TPAL: 170618880 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.032 (1461611770032127) C=3901: Process 628 increments ProvideMemoryCounter: 1543485 -> 1543486

T=2016-04-25 16:16:10.033 (1461611770033479) C=1030: Process 628 increments ProvideMemoryCounter: 1543486 -> 1543487

T=2016-04-25 16:16:10.033 (1461611770033485) C=1030: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.033 (1461611770033485) C=1030: TPAL=8932638720, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.033 (1461611770033486) C=1030: Process 628 reserves all memory until TPAL: 97005568 (->97005568), PM-inx=0

T=2016-04-25 16:16:10.036 (1461611770036609) C=1030: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.036 (1461611770036610) C=1030: Process 628 reserves all memory until TPAL: 171429888 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.036 (1461611770036614) C=1030: Process 628 increments ProvideMemoryCounter: 1543487 -> 1543488

T=2016-04-25 16:16:10.049 (1461611770049384) C=4028: Process 7008 increments ProvideMemoryCounter: 1543488 -> 1543489

T=2016-04-25 16:16:10.049 (1461611770049392) C=4028: ==== Process 7008 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.629 (1461611770629267) C=4034: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.629 (1461611770629267) C=4034: TPAL=8927178752, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.629 (1461611770629267) C=4034: Process 628 reserves all memory until TPAL: 86671360 (->86671360), PM-inx=0

T=2016-04-25 16:16:10.631 (1461611770631065) C=4034: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.631 (1461611770631066) C=4034: Process 628 reserves all memory until TPAL: 1048576 (->87719936), PM-inx=0

T=2016-04-25 16:16:10.631 (1461611770631191) C=4034: Process 628 requests process 966 to reserve=50098176 and to shrink=50098176

T=2016-04-25 16:16:10.633 (1461611770633509) C=4034: Process 628 requests process 968 to reserve=21975040 and to shrink=21975040

T=2016-04-25 16:16:10.633 (1461611770633986) C=4034: Process 628 requests process 995 to reserve=180715520 and to shrink=702640128

T=2016-04-25 16:16:10.784 (1461611770784140) C=1014: Process 995 incremented IPMM reserved bytes by 131072 (->131072) - reason: shift to IPMM reservation.

T=2016-04-25 16:16:10.784 (1461611770784250) C=4034: Process 628 reserves all memory until TPAL: 6983680 (->94703616), PM-inx=0

T=2016-04-25 16:16:10.784 (1461611770784254) C=4034: Process 628 uses IPMM reserved bytes (131072) to increase reservation (-> 94834688) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.784 (1461611770784256) C=4034: Process 628 requests process 1272 to reserve=28815360 and to shrink=28815360

T=2016-04-25 16:16:10.784 (1461611770784828) C=4034: Process 628 requests process 7008 to reserve=173600768 and to shrink=1629143040

T=2016-04-25 16:16:10.874 (1461611770874990) C=4034: Process 628 requested self compaction, to reserve=173600768, to shrink=189235200: Reserved 173600768!

T=2016-04-25 16:16:10.874 (1461611770874991) C=4034: Process 628 incremented IPMM reserved bytes by 267386880 (->267386880) - reason: compact own process (shrink request: 189235200).

T=2016-04-25 16:16:10.874 (1461611770874994) C=4034: Process 628 uses IPMM reserved bytes (267386880) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:10.874 (1461611770874999) C=4034: Process 628 increments ProvideMemoryCounter: 1543527 -> 1543528

T=2016-04-25 16:16:10.875 (1461611770875026) C=4038: Process 628 increments ProvideMemoryCounter: 1543528 -> 1543529

T=2016-04-25 16:16:10.875 (1461611770875031) C=4038: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.875 (1461611770875031) C=4038: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.875 (1461611770875032) C=4038: Process 628 reserves all memory until TPAL: 93786112 (->93786112), PM-inx=0

T=2016-04-25 16:16:10.877 (1461611770877046) C=4038: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.877 (1461611770877047) C=4038: Process 628 reserves all memory until TPAL: 174649344 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.877 (1461611770877049) C=4038: Process 628 increments ProvideMemoryCounter: 1543529 -> 1543530

T=2016-04-25 16:16:10.877 (1461611770877075) C=1029: Process 628 increments ProvideMemoryCounter: 1543530 -> 1543531

T=2016-04-25 16:16:10.877 (1461611770877079) C=1029: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.877 (1461611770877079) C=1029: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.877 (1461611770877080) C=1029: Process 628 reserves all memory until TPAL: 268435456 (->268435456), PM-inx=0

T=2016-04-25 16:16:10.877 (1461611770877082) C=1029: Process 628 increments ProvideMemoryCounter: 1543531 -> 1543532

T=2016-04-25 16:16:10.877 (1461611770877102) C=3903: Process 628 increments ProvideMemoryCounter: 1543532 -> 1543533

T=2016-04-25 16:16:10.877 (1461611770877106) C=3903: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:10.877 (1461611770877106) C=3903: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:10.877 (1461611770877107) C=3903: Process 628 reserves all memory until TPAL: 92332032 (->92332032), PM-inx=0

T=2016-04-25 16:16:10.879 (1461611770879340) C=3903: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:10.879 (1461611770879341) C=3903: Process 628 reserves all memory until TPAL: 1048576 (->93380608), PM-inx=0

T=2016-04-25 16:16:10.879 (1461611770879491) C=3903: Process 628 requests process 966 to reserve=49991680 and to shrink=49991680

T=2016-04-25 16:16:10.880 (1461611770880371) C=3903: Process 628 requests process 968 to reserve=21925888 and to shrink=21925888

T=2016-04-25 16:16:10.880 (1461611770880729) C=3903: Process 628 requests process 995 to reserve=175054848 and to shrink=701145088

T=2016-04-25 16:16:11.028 (1461611771028387) C=3903: Process 628 reserves all memory until TPAL: 2502656 (->95883264), PM-inx=0

T=2016-04-25 16:16:11.028 (1461611771028394) C=3903: Process 628 requests process 1272 to reserve=28753920 and to shrink=28753920

T=2016-04-25 16:16:11.028 (1461611771028848) C=3903: Process 628 requests process 7008 to reserve=172552192 and to shrink=1625677824

T=2016-04-25 16:16:11.113 (1461611771113087) C=3903: Process 628 requested self compaction, to reserve=172552192, to shrink=188833792: Reserved 172552192!

T=2016-04-25 16:16:11.113 (1461611771113088) C=3903: Process 628 incremented IPMM reserved bytes by 268435456 (->268435456) - reason: compact own process (shrink request: 188833792).

T=2016-04-25 16:16:11.113 (1461611771113091) C=3903: Process 628 uses IPMM reserved bytes (268435456) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:11.113 (1461611771113097) C=3903: Process 628 increments ProvideMemoryCounter: 1543533 -> 1543534

T=2016-04-25 16:16:11.113 (1461611771113120) C=4034: Process 628 increments ProvideMemoryCounter: 1543534 -> 1543535

T=2016-04-25 16:16:11.113 (1461611771113125) C=4034: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:11.113 (1461611771113125) C=4034: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:11.113 (1461611771113126) C=4034: Process 628 reserves all memory until TPAL: 268435456 (->268435456), PM-inx=0

T=2016-04-25 16:16:11.113 (1461611771113129) C=4034: Process 628 increments ProvideMemoryCounter: 1543535 -> 1543536

T=2016-04-25 16:16:11.113 (1461611771113160) C=637: Process 628 increments ProvideMemoryCounter: 1543536 -> 1543537

T=2016-04-25 16:16:11.113 (1461611771113164) C=637: ==== Process 628 must provide memory. Bytes to reserve=4096, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:11.113 (1461611771113165) C=637: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:11.113 (1461611771113165) C=637: Process 628 reserves all memory until TPAL: 4096 (->4096), PM-inx=0

T=2016-04-25 16:16:11.113 (1461611771113167) C=637: Process 628 increments ProvideMemoryCounter: 1543537 -> 1543538

T=2016-04-25 16:16:11.113 (1461611771113569) C=1031: Process 628 increments ProvideMemoryCounter: 1543538 -> 1543539

T=2016-04-25 16:16:11.113 (1461611771113573) C=1031: ==== Process 628 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:16:11.113 (1461611771113573) C=1031: TPAL=8927309824, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:16:11.113 (1461611771113573) C=1031: Process 628 reserves all memory until TPAL: 95879168 (->95879168), PM-inx=0

T=2016-04-25 16:16:11.114 (1461611771114752) C=1031: Process 628 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:16:11.114 (1461611771114753) C=1031: Process 628 reserves all memory until TPAL: 4096 (->95883264), PM-inx=0

T=2016-04-25 16:16:11.114 (1461611771114892) C=1031: Process 628 requests process 7008 to reserve=172552192 and to shrink=1624145920

T=2016-04-25 16:16:11.205 (1461611771205237) C=1031: Process 628 requested self compaction, to reserve=172552192, to shrink=188653568: Reserved 172552192!

T=2016-04-25 16:16:11.205 (1461611771205239) C=1031: Process 628 incremented IPMM reserved bytes by 535011328 (->535011328) - reason: compact own process (shrink request: 188653568).

T=2016-04-25 16:16:11.205 (1461611771205242) C=1031: Process 628 uses IPMM reserved bytes (535011328) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:16:11.205 (1461611771205249) C=1031: Process 628 increments ProvideMemoryCounter: 1543539 ->

T=2016-04-25 16:17:27.617 (1461611847617200) C=7271: Process 7008 requested self compaction, to reserve=260755456, to shrink=926453760: Reserved 260755456!

T=2016-04-25 16:17:27.901 (1461611847901216) C=7014: Process 7008 increments ProvideMemoryCounter: 1545534 -> 1545535

T=2016-04-25 16:17:27.901 (1461611847901220) C=7014: ==== Process 7008 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:17:28.005 (1461611848005506) C=7014: Process 7008 requested self compaction, to reserve=262852608, to shrink=927748096: Reserved 262852608!

T=2016-04-25 16:17:28.005 (1461611848005509) C=7014: Process 7008 incremented IPMM reserved bytes by 264241152 (->264241152) - reason: compact own process (shrink request: 927748096).

T=2016-04-25 16:17:28.005 (1461611848005516) C=7014: Process 7008 uses IPMM reserved bytes (264241152) to increase reservation (-> 268435456) and decrease sum allocation limits by 0.

T=2016-04-25 16:17:33.353 (1461611853353422) C=4057: Process 7008 requests process 995 to reserve=1601536 and to shrink=1522708480

T=2016-04-25 16:17:33.382 (1461611853382918) C=4057: Process 7008 requests process 1272 to reserve=1601536 and to shrink=62443520

T=2016-04-25 16:17:33.609 (1461611853609818) C=4057: Process 7008 requested self compaction, to reserve=1601536, to shrink=3560169472: Reserved 0!

T=2016-04-25 16:17:33.609 (1461611853609828) C=4057: Process 7008 requests process 628 to reserve=1601536 and to shrink=380964864

T=2016-04-25 16:17:33.615 (1461611853615595) C=4057: Process 7008 requests process 966 to reserve=1601536 and to shrink=108580864

T=2016-04-25 16:17:33.616 (1461611853616092) C=4057: Last chance! Size to provide=4611686018427383808, old precharge=266833920, current precharge=266833920, iter=4

T=2016-04-25 16:17:33.616 (1461611853616336) C=4057: Process 7008 requests process 1272 to reserve=1601536 and to shrink=4611686018427383808

T=2016-04-25 16:17:33.838 (1461611853838413) C=4057: Process 7008 requested self compaction, to reserve=1601536, to shrink=4611686018427383808: Reserved 0!

T=2016-04-25 16:17:33.838 (1461611853838423) C=4057: Process 7008 requests process 628 to reserve=1601536 and to shrink=4611686018427383808

T=2016-04-25 16:17:33.843 (1461611853843112) C=4057: Process 7008 requests process 966 to reserve=1601536 and to shrink=4611686018427383808

T=2016-04-25 16:17:33.843 (1461611853843428) C=4057: Process 7008 requests process 968 to reserve=1601536 and to shrink=4611686018427383808

T=2016-04-25 16:17:33.843 (1461611853843728) C=4057: Process 7008 requests process 995 to reserve=1601536 and to shrink=4611686018427383808

T=2016-04-25 16:17:33.869 (1461611853869435) C=4057: Process 7008 updates last failed ProvideMemory: free size=266833920, ?global-problem?=1, PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869435) C=4057: Process 7008 increments ProvideMemoryCounter: 1545605 -> 1545606

T=2016-04-25 16:17:33.869 (1461611853869435) C=4057: ! ! ! >>>> IPMM could not provide memory, could only reserve=266833920! ?only-TPAL-increase?=0. Cached sum TPAL=149034151936. ?reduce GAL?=0. <<<< ! ! !

T=2016-04-25 16:17:33.869 (1461611853869490) C=4057: SlotInfo: pid=628, AL=8292470784, U=6334672021, TPAL=8292470784

T=2016-04-25 16:17:33.869 (1461611853869516) C=4057: SlotInfo: pid=966, AL=1143779328, U=324837617, TPAL=2363437056

T=2016-04-25 16:17:33.869 (1461611853869536) C=4057: SlotInfo: pid=968, AL=1036574720, U=227880645, TPAL=1036574720

T=2016-04-25 16:17:33.869 (1461611853869561) C=4057: SlotInfo: pid=995, AL=33145008128, U=29186203111, TPAL=33145008128

T=2016-04-25 16:17:33.869 (1461611853869582) C=4057: SlotInfo: pid=1272, AL=1359212544, U=527036910, TPAL=1359212544

T=2016-04-25 16:17:33.869 (1461611853869583) C=4057: SlotInfo: pid=7008, AL=77494628352, U=72005726185, TPAL=77494628352

T=2016-04-25 16:17:33.869 (1461611853869583) C=4057: GAL=149034151936, IPMM reservation=0, SHARED_MEMORY=23298207744, CODE_SIZE=1742049280.

T=2016-04-25 16:17:33.869 (1461611853869627) C=3870: Process 7008 increments ProvideMemoryCounter: 1545606 -> 1545607

T=2016-04-25 16:17:33.869 (1461611853869633) C=3870: ==== Process 7008 must provide memory. Bytes to reserve=268435456, to shrink=0, ?only-TPAL-increase?=0. ====

T=2016-04-25 16:17:33.869 (1461611853869633) C=3870: TPAL=77494628352, PAL=149034151936, GAL=148899934208, cached sum TPAL=149034151936.

T=2016-04-25 16:17:33.869 (1461611853869634) C=3870: Process 7008 reserves all memory until TPAL: 266833920 (->266833920), PM-inx=0

T=2016-04-25 16:17:33.869 (1461611853869710) C=4044: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869710) C=4044: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.869 (1461611853869713) C=4040: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869714) C=4040: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.869 (1461611853869722) C=4052: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869723) C=4052: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.869 (1461611853869728) C=4051: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869732) C=4051: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.869 (1461611853869731) C=4050: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.869 (1461611853869732) C=4050: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.876 (1461611853876526) C=1036: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.876 (1461611853876526) C=1036: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.876 (1461611853876537) C=3981: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.876 (1461611853876537) C=3981: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.876 (1461611853876545) C=14192: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.876 (1461611853876546) C=14192: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.876 (1461611853876550) C=3959: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.876 (1461611853876551) C=3959: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.877 (1461611853877584) C=3870: Process 7008 requested defragmentation of other processes, providing 0.

T=2016-04-25 16:17:33.877 (1461611853877720) C=3870: Process 7008 requests process 968 to reserve=1601536 and to shrink=10256384

T=2016-04-25 16:17:33.877 (1461611853877733) C=7271: Process 7008: early exit, PM-inx before requesting lock=1545599, last failed PM-inx=1545605

T=2016-04-25 16:17:33.877 (1461611853877734) C=7271: Process 7008 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.878 (1461611853878490) C=3991: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.878 (1461611853878491) C=3991: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.879 (1461611853879801) C=1289: Process 1272: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.879 (1461611853879803) C=1289: Process 1272 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.881 (1461611853881720) C=3969: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.881 (1461611853881721) C=3969: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.886 (1461611853886781) C=3989: Process 995: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.886 (1461611853886782) C=3989: Process 995 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.889 (1461611853889241) C=3870: Process 7008 requests process 995 to reserve=1601536 and to shrink=327868416

T=2016-04-25 16:17:33.944 (1461611853944626) C=7270: Process 7008: early exit, PM-inx before requesting lock=1545581, last failed PM-inx=1545605

T=2016-04-25 16:17:33.944 (1461611853944626) C=7270: Process 7008 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:33.952 (1461611853952386) C=645: Process 628: early exit, PM-inx before requesting lock=1545555, last failed PM-inx=1545605

T=2016-04-25 16:17:33.952 (1461611853952387) C=645: Process 628 (early exit): To reserve=268435456, free size for last failed=266833920

T=2016-04-25 16:17:34.003 (1461611854003877) C=1014: Process 995 incremented IPMM reserved bytes by 540672 (->540672) - reason: shift to IPMM reservation.

T=2016-04-25 16:17:34.004 (1461611854004187) C=3870: Process 7008 uses IPMM reserved bytes (540672) to increase reservation (-> 267374592) and decrease sum allocation limits by 0.

T=2016-04-25 16:17:34.004 (1461611854004190) C=3870: Process 7008 requests process 1272 to reserve=1060864 and to shrink=13447168

[STAT] Statistics data:

M_COMPACTION_THREAD - (id= 1, type= public:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= Default)

"NUM_COMPACTION_COLLISIONS";"NUM_COMPACTIONS";"LAST_SIZE_COMPACTION_REQUESTS [byte]";"MAX_SIZE_COMPACTION_REQUESTS [byte]";"MIN_SIZE_COMPACTION_REQUESTS [byte]";"SUM_SIZE_COMPACTION_REQUESTS [byte]";"AVG_SIZE_COMPACTION_REQUESTS [byte]";"LAST_SIZE_FREEABLE [byte]";"MAX_SIZE_FREEABLE [byte]";"MIN_SIZE_FREEABLE [byte]";"SUM_SIZE_FREEABLE [byte]";"AVG_SIZE_FREEABLE [byte]";"LAST_SIZE_FREED_BY_GARBAGE_COLLECTION [byte]";"MAX_SIZE_FREED_BY_GARBAGE_COLLECTION [byte]";"MIN_SIZE_FREED_BY_GARBAGE_COLLECTION [byte]";"SUM_SIZE_FREED_BY_GARBAGE_COLLECTION [byte]";"AVG_SIZE_FREED_BY_GARBAGE_COLLECTION [byte]";"LAST_COMPACTION_RESULT [byte]";"MAX_COMPACTION_RESULT [byte]";"MIN_COMPACTION_RESULT [byte]";"SUM_COMPACTION_RESULT [byte]";"AVG_COMPACTION_RESULT [byte]"

0;538;13447168;4611686018427383808;13422592;9223372049685475328;17568327713686619;0;0;0;0;0;0;268435456;0;3238559744;6019627;1060864;1125720064;126976;22213091328;41288273

(M_COMPACTION_THREAD, 1 rows in 33us)

M_DEV_IPMM - (id= 12, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"GLOBAL_ALLOCATION_LIMIT [byte]";"ASYNC_FREE_THRESHOLD [byte]";"ASYNC_FREE_TARGET [byte]";"CACHED_SHARED_MEMORY [byte]";"SHARED_MEMORY_TEMP_LIMIT [byte]";"OOM_RESERVATION_ALLOCATOR_SIZE [byte]";"OOM_RESERVATION_ALLOCATOR_TEMP_LIMIT [byte]";"ALLOCATION_LIMIT_SUM [byte]";"CACHED_ALLOCATION_LIMIT_SUM [byte]";"WEIGHT_SUM";"CACHED_WEIGHT_SUM";"NUM_CHECKS";"TERMINATED_PROCESSES"

149034151936;100;95;23298207744;23298207744;302563328;302563328;149034151936;149034151936;600;600;9668962;0

(M_DEV_IPMM, 1 rows in 48us)

M_DEV_IPMM_SLOTS - (id= 13, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"SLOT_ID";"PROCESS_IS_ALIVE";"PROCESS_ID";"CREATION_TIME";"COMPACTORS_ACTIVE";"CURRENT_ALLOCATED";"PROCESS_ALLOCATION_LIMIT";"TEMP_PROCESS_ALLOCATION_LIMIT";"NUM_THREADS_WAITING_FOR_IPMM_LOCK";"EXECUTES_COMPACTION";"REQUESTS_COMPACTION"

0;1;628;300674672;1;8292470784;149034151936;8292470784;8;0;0

1;1;966;300675561;1;1143779328;149034151936;2363437056;0;0;0

2;1;968;300675562;1;1036574720;149034151936;1036574720;2;0;0

3;1;995;300675884;1;33144467456;149034151936;33144467456;6;0;0

4;1;1272;300682944;1;1359212544;149034151936;1359212544;2;0;0

5;1;7008;300981831;1;77496229888;149034151936;77495169024;3;0;1

(M_DEV_IPMM_SLOTS, 6 rows in 654us)

M_DEV_MEMORY_OBJECT_CONTAINER - (id= 29, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"LAST_SIZE_BEFORE_SHRINK [byte]";"MAX_SIZE_BEFORE_SHRINK [byte]";"MIN_SIZE_BEFORE_SHRINK [byte]";"SUM_SIZE_BEFORE_SHRINK [byte]";"AVG_SIZE_BEFORE_SHRINK [byte]";"LAST_SIZE_AFTER_SHRINK [byte]";"MAX_SIZE_AFTER_SHRINK [byte]";"MIN_SIZE_AFTER_SHRINK [byte]";"SUM_SIZE_AFTER_SHRINK [byte]";"AVG_SIZE_AFTER_SHRINK [byte]";"LAST_SIZE_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_RELEASED_BY_SHRINK [byte]";"LAST_SIZE_PAGE_LOADABLE_COLUMNS_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_PAGE_LOADABLE_COLUMNS_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_PAGE_LOADABLE_COLUMNS_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_PAGE_LOADABLE_COLUMNS_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_PAGE_LOADABLE_COLUMNS_RELEASED_BY_SHRINK [byte]";"LAST_SIZE_EARLY_UNLOAD_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_EARLY_UNLOAD_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_EARLY_UNLOAD_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_EARLY_UNLOAD_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_EARLY_UNLOAD_RELEASED_BY_SHRINK [byte]";"LAST_SIZE_SHORT_TERM_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_SHORT_TERM_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_SHORT_TERM_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_SHORT_TERM_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_SHORT_TERM_RELEASED_BY_SHRINK [byte]";"LAST_SIZE_MID_TERM_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_MID_TERM_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_MID_TERM_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_MID_TERM_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_MID_TERM_RELEASED_BY_SHRINK [byte]";"LAST_SIZE_LONG_TERM_RELEASED_BY_SHRINK [byte]";"MAX_SIZE_LONG_TERM_RELEASED_BY_SHRINK [byte]";"MIN_SIZE_LONG_TERM_RELEASED_BY_SHRINK [byte]";"SUM_SIZE_LONG_TERM_RELEASED_BY_SHRINK [byte]";"AVG_SIZE_LONG_TERM_RELEASED_BY_SHRINK [byte]";"LAST_NUM_OBJECTS_RELEASED_BY_SHRINK";"MAX_NUM_OBJECTS_RELEASED_BY_SHRINK";"MIN_NUM_OBJECTS_RELEASED_BY_SHRINK";"SUM_NUM_OBJECTS_RELEASED_BY_SHRINK";"AVG_NUM_OBJECTS_RELEASED_BY_SHRINK";"LAST_NUM_PAGE_LOADABLE_COLUMNS_OBJECTS_RELEASED_BY_SHRINK";"MAX_NUM_PAGE_LOADABLE_COLUMNS_OBJECTS_RELEASED_BY_SHRINK";"MIN_NUM_PAGE_LOADABLE_COLUMNS_OBJECTS_RELEASED_BY_SHRINK";"SUM_NUM_PAGE_LOADABLE_COLUMNS_OBJECTS_RELEASED_BY_SHRINK";"AVG_NUM_PAGE_LOADABLE_COLUMNS_OBJECTS_RELEASED_BY_SHRINK";"LAST_NUM_EARLY_UNLOAD_RELEASED_BY_SHRINK";"MAX_NUM_EARLY_UNLOAD_RELEASED_BY_SHRINK";"MIN_NUM_EARLY_UNLOAD_RELEASED_BY_SHRINK";"SUM_NUM_EARLY_UNLOAD_RELEASED_BY_SHRINK";"AVG_NUM_EARLY_UNLOAD_RELEASED_BY_SHRINK";"LAST_NUM_SHORT_TERM_RELEASED_BY_SHRINK";"MAX_NUM_SHORT_TERM_RELEASED_BY_SHRINK";"MIN_NUM_SHORT_TERM_RELEASED_BY_SHRINK";"SUM_NUM_SHORT_TERM_RELEASED_BY_SHRINK";"AVG_NUM_SHORT_TERM_RELEASED_BY_SHRINK";"LAST_NUM_MID_TERM_RELEASED_BY_SHRINK";"MAX_NUM_MID_TERM_RELEASED_BY_SHRINK";"MIN_NUM_MID_TERM_RELEASED_BY_SHRINK";"SUM_NUM_MID_TERM_RELEASED_BY_SHRINK";"AVG_NUM_MID_TERM_RELEASED_BY_SHRINK";"LAST_NUM_LONG_TERM_RELEASED_BY_SHRINK";"MAX_NUM_LONG_TERM_RELEASED_BY_SHRINK";"MIN_NUM_LONG_TERM_RELEASED_BY_SHRINK";"SUM_NUM_LONG_TERM_RELEASED_BY_SHRINK";"AVG_NUM_LONG_TERM_RELEASED_BY_SHRINK";"LAST_SIZE_TO_FREE_BY_SHRINK [byte]";"MAX_SIZE_TO_FREE_BY_SHRINK [byte]";"MIN_SIZE_TO_FREE_BY_SHRINK [byte]";"SUM_SIZE_TO_FREE_BY_SHRINK [byte]";"AVG_SIZE_TO_FREE_BY_SHRINK [byte]";"LAST_SHRINK_RUN_TIME [us]";"MAX_SHRINK_RUN_TIME [us]";"MIN_SHRINK_RUN_TIME [us]";"SUM_SHRINK_RUN_TIME [us]";"AVG_SHRINK_RUN_TIME [us]";"COUNT_RESIZE_YIELD";"COUNT_REMOVE_HEADER_YIELD";"COUNT_PUT_HEADER_YIELD";"COUNT_EXECUTED_SHRINKS";"COUNT_SKIPPED_SHRINKS";"COUNT_SHRINK_GET_FREE_RESTARTS";"SIZE_MANAGED_RESOURCES [byte]";"SIZE_MANAGED_NON_SWAPPABLE_RESOURCES [byte]";"SIZE_MANAGED_PAGE_LOADABLE_COLUMNS_RESOURCES [byte]";"PAGE_LOADABLE_COLUMNS_MIN_SIZE [byte]";"PAGE_LOADABLE_COLUMNS_LIMIT [byte]";"UNUSED_RETENTION_PERIOD [s]";"UNUSED_RETENTION_PERIOD_CHECKS [s]";"UNLOAD_LOWER_BOUND [byte]"

(M_DEV_MEMORY_OBJECT_CONTAINER, 0 rows in 64us)

M_DEV_PROCESS_MEMORY - (id= 35, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"PROCESS_ALLOCATION_LIMIT [byte]";"CURRENT_ALLOCATED_SIZE [byte]";"PAGEMAP_SIZE [byte]";"BIGBLOCK_SEGMENT_SIZE [byte]";"BIGBLOCK_FREE_SIZE [byte]";"COMPACTION_COUNT";"ILLEGAL_HEAP_MEMORY_ALLOCATIONS";"HEAP_MEMORY_FAILURES";"HEAP_MEMORY_LIMIT_FAILURES";"STACK_MEMORY_FAILURES";"SHARED_MEMORY_LIMIT_FAILURES";"RAM_SIZE [byte]";"ALLOCATED_SYSTEM_MEMORY [byte]";"DEALLOCATED_SYSTEM_MEMORY [byte]";"CURRENT_ALLOCATED_SYSTEM_MEMORY [byte]";"TEMP_PROCESS_ALLOCATION_LIMIT [byte]";"LAST_ALLOCATION_PREPARATION_DURATION [ticks]";"MAX_ALLOCATION_PREPARATION_DURATION [ticks]";"MIN_ALLOCATION_PREPARATION_DURATION [ticks]";"SUM_ALLOCATION_PREPARATION_DURATION [ticks]";"AVG_ALLOCATION_PREPARATION_DURATION [ticks]";"STATEMENT_MEMORY_LIMIT_THRESHOLD [%]";"GC_UNUSED_MEMORY_THRESHOLD_REL [%]";"GC_UNUSED_MEMORY_THRESHOLD_ABS [byte]"

149034151936;1359212544;28311552;268435456;0;10;0;2;0;0;0;158630682624;13436020899840;13435439710208;581189632;1359212544;0;41904701;0;995971371;3221;0;-1;0

(M_DEV_PROCESS_MEMORY, 1 rows in 23us)

M_DEV_SYSTEM_MEMORY_FAILURES - (id= 38, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"MEM_DECOMMIT_SYSTEM_PAGES_FAILURES";"MEM_RESERVE_SYSTEM_PAGES_FAILURES";"MEM_ALLOC_SYSTEM_PAGES_FAILURES";"MEM_FREE_SYSTEM_PAGES_FAILURES";"MEM_FREE_SYSTEM_PAGES_FAILURE_SIZE [byte]";"MEM_FREE_DECOMMITTED_SYSTEM_PAGES_FAILURE_SIZE [byte]";"GET_USED_SHARED_MEMORY_FAILURES";"MEM_PROTECT_FAILURES"

0;0;0;0;0;0;0;0

(M_DEV_SYSTEM_MEMORY_FAILURES, 1 rows in 28us)

M_DEV_SYSTEM_MEMORY_PROVIDER - (id= 39, type= dev:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"SIZE_ALLOCATED [byte]";"COUNT_ALLOCATIONS";"COUNT_BIG_ALLOCATIONS";"COUNT_ALLOCATION_ERRORS";"SIZE_OVERHEAD [byte]";"SIZE_RESERVED_LIMITED [byte]";"SIZE_RESERVED_UNLIMITED [byte]";"SIZE_UNRESERVED [byte]";"SIZE_RESERVED_OVERHEAD [byte]";"SIZE_COMMITTED [byte]"

46094192;702;1;0;3216;16777216;54693888;0;0;40894464

(M_DEV_SYSTEM_MEMORY_PROVIDER, 1 rows in 19us)

M_VOLUME_IO_TOTAL_STATISTICS_ - (id= 68, type= private:remote:multi, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"PATH";"FILESYSTEM_TYPE";"TYPE";"CONFIGURATION";"OPEN_CALL_COUNT";"CLOSE_CALL_COUNT";"EXISTS_CALL_COUNT";"REMOVE_CALL_COUNT";"RESIZE_CALL_COUNT";"SYNC_CALL_COUNT";"GETSIZE_CALL_COUNT";"GETMODIFICATIONTIME_CALL_COUNT";"GETDEVICEID_CALL_COUNT";"BLOCKED_WRITE_REQUESTS";"MAX_BLOCKED_WRITE_REQUESTS";"EOF_READS";"TOTAL_READS";"TOTAL_TRIGGER_ASYNC_READS";"ACTIVE_ASYNC_READS_COUNT";"TRIGGER_READ_RATIO";"TOTAL_SHORT_READS";"TOTAL_FULL_RETRY_READS";"TOTAL_FAILED_READS";"TOTAL_READ_SIZE [byte]";"TOTAL_READ_TIME [us]";"TOTAL_APPENDS";"TOTAL_WRITES";"TOTAL_TRIGGER_ASYNC_WRITES";"ACTIVE_ASYNC_WRITES_COUNT";"TRIGGER_WRITE_RATIO";"TOTAL_SHORT_WRITES";"TOTAL_FULL_RETRY_WRITES";"TOTAL_FAILED_WRITES";"TOTAL_WRITE_SIZE [byte]";"TOTAL_WRITE_TIME [us]";"TOTAL_IO_TIME [us]"

(M_VOLUME_IO_TOTAL_STATISTICS_, 0 rows in 30us)

M_DEV_VOLUME_IO_DETAILED_STATISTICS_ - (id= 44, type= private:remote:multi, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"PATH";"FILESYSTEM_TYPE";"TYPE";"CONFIGURATION";"MAX_IO_BUFFER_SIZE [byte]";"APPEND_COUNT";"ACTIVE_APPEND_COUNT";"LAST_APPEND_SIZE [byte]";"MAX_APPEND_SIZE [byte]";"MIN_APPEND_SIZE [byte]";"SUM_APPEND_SIZE [byte]";"AVG_APPEND_SIZE [byte]";"LAST_APPEND_TIME [us]";"MAX_APPEND_TIME [us]";"MIN_APPEND_TIME [us]";"SUM_APPEND_TIME [us]";"AVG_APPEND_TIME [us]";"FAILED_APPEND_COUNT";"WRITE_COUNT";"ACTIVE_WRITE_COUNT";"LAST_WRITE_SIZE [byte]";"MAX_WRITE_SIZE [byte]";"MIN_WRITE_SIZE [byte]";"SUM_WRITE_SIZE [byte]";"AVG_WRITE_SIZE [byte]";"LAST_WRITE_TIME [us]";"MAX_WRITE_TIME [us]";"MIN_WRITE_TIME [us]";"SUM_WRITE_TIME [us]";"AVG_WRITE_TIME [us]";"TRIGGER_ASYNC_WRITE_COUNT";"ACTIVE_TRIGGER_ASYNC_WRITE_COUNT";"LAST_TRIGGER_ASYNC_WRITE_SIZE [byte]";"MAX_TRIGGER_ASYNC_WRITE_SIZE [byte]";"MIN_TRIGGER_ASYNC_WRITE_SIZE [byte]";"SUM_TRIGGER_ASYNC_WRITE_SIZE [byte]";"AVG_TRIGGER_ASYNC_WRITE_SIZE [byte]";"LAST_TRIGGER_ASYNC_WRITE_TIME [us]";"MAX_TRIGGER_ASYNC_WRITE_TIME [us]";"MIN_TRIGGER_ASYNC_WRITE_TIME [us]";"SUM_TRIGGER_ASYNC_WRITE_TIME [us]";"AVG_TRIGGER_ASYNC_WRITE_TIME [us]";"LAST_TRIGGER_ASYNC_WRITE_TRIGGER_TIME [us]";"MAX_TRIGGER_ASYNC_WRITE_TRIGGER_TIME [us]";"MIN_TRIGGER_ASYNC_WRITE_TRIGGER_TIME [us]";"SUM_TRIGGER_ASYNC_WRITE_TRIGGER_TIME [us]";"AVG_TRIGGER_ASYNC_WRITE_TRIGGER_TIME [us]";"TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_COUNT";"LAST_TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_TIME [us]";"MAX_TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_TIME [us]";"MIN_TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_TIME [us]";"SUM_TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_TIME [us]";"AVG_TRIGGER_ASYNC_WRITE_SYNC_SUBMIT_TIME [us]";"TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_COUNT";"LAST_TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_TIME [us]";"MAX_TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_TIME [us]";"MIN_TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_TIME [us]";"SUM_TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_TIME [us]";"AVG_TRIGGER_ASYNC_WRITE_ASYNC_SUBMIT_TIME [us]";"LAST_TRIGGER_ASYNC_WRITE_CALLBACK_TIME [us]";"MAX_TRIGGER_ASYNC_WRITE_CALLBACK_TIME [us]";"MIN_TRIGGER_ASYNC_WRITE_CALLBACK_TIME [us]";"SUM_TRIGGER_ASYNC_WRITE_CALLBACK_TIME [us]";"AVG_TRIGGER_ASYNC_WRITE_CALLBACK_TIME [us]";"SHORT_WRITE_COUNT";"REQUESTS_WITH_SHORT_WRITES_COUNT";"AVG_SHORT_WRITES_PER_REQUEST_COUNT";"MAX_SHORT_WRITES_PER_REQUEST_COUNT";"FULL_RETRY_WRITE_COUNT";"REQUESTS_WITH_FULL_RETRY_WRITES_COUNT";"AVG_FULL_RETRY_WRITES_PER_REQUEST_COUNT";"MAX_FULL_RETRY_WRITES_PER_REQUEST_COUNT";"FAILED_WRITE_COUNT";"READ_COUNT";"ACTIVE_READ_COUNT";"LAST_READ_SIZE [byte]";"MAX_READ_SIZE [byte]";"MIN_READ_SIZE [byte]";"SUM_READ_SIZE [byte]";"AVG_READ_SIZE [byte]";"LAST_READ_TIME [us]";"MAX_READ_TIME [us]";"MIN_READ_TIME [us]";"SUM_READ_TIME [us]";"AVG_READ_TIME [us]";"TRIGGER_ASYNC_READ_COUNT";"ACTIVE_TRIGGER_ASYNC_READ_COUNT";"LAST_TRIGGER_ASYNC_READ_SIZE [byte]";"MAX_TRIGGER_ASYNC_READ_SIZE [byte]";"MIN_TRIGGER_ASYNC_READ_SIZE [byte]";"SUM_TRIGGER_ASYNC_READ_SIZE [byte]";"AVG_TRIGGER_ASYNC_READ_SIZE [byte]";"LAST_TRIGGER_ASYNC_READ_TIME [us]";"MAX_TRIGGER_ASYNC_READ_TIME [us]";"MIN_TRIGGER_ASYNC_READ_TIME [us]";"SUM_TRIGGER_ASYNC_READ_TIME [us]";"AVG_TRIGGER_ASYNC_READ_TIME [us]";"LAST_TRIGGER_ASYNC_READ_TRIGGER_TIME [us]";"MAX_TRIGGER_ASYNC_READ_TRIGGER_TIME [us]";"MIN_TRIGGER_ASYNC_READ_TRIGGER_TIME [us]";"SUM_TRIGGER_ASYNC_READ_TRIGGER_TIME [us]";"AVG_TRIGGER_ASYNC_READ_TRIGGER_TIME [us]";"TRIGGER_ASYNC_READ_SYNC_SUBMIT_COUNT";"LAST_TRIGGER_ASYNC_READ_SYNC_SUBMIT_TIME [us]";"MAX_TRIGGER_ASYNC_READ_SYNC_SUBMIT_TIME [us]";"MIN_TRIGGER_ASYNC_READ_SYNC_SUBMIT_TIME [us]";"SUM_TRIGGER_ASYNC_READ_SYNC_SUBMIT_TIME [us]";"AVG_TRIGGER_ASYNC_READ_SYNC_SUBMIT_TIME [us]";"TRIGGER_ASYNC_READ_ASYNC_SUBMIT_COUNT";"LAST_TRIGGER_ASYNC_READ_ASYNC_SUBMIT_TIME [us]";"MAX_TRIGGER_ASYNC_READ_ASYNC_SUBMIT_TIME [us]";"MIN_TRIGGER_ASYNC_READ_ASYNC_SUBMIT_TIME [us]";"SUM_TRIGGER_ASYNC_READ_ASYNC_SUBMIT_TIME [us]";"AVG_TRIGGER_ASYNC_READ_ASYNC_SUBMIT_TIME [us]";"LAST_TRIGGER_ASYNC_READ_CALLBACK_TIME [us]";"MAX_TRIGGER_ASYNC_READ_CALLBACK_TIME [us]";"MIN_TRIGGER_ASYNC_READ_CALLBACK_TIME [us]";"SUM_TRIGGER_ASYNC_READ_CALLBACK_TIME [us]";"AVG_TRIGGER_ASYNC_READ_CALLBACK_TIME [us]";"SHORT_READ_COUNT";"REQUESTS_WITH_SHORT_READS_COUNT";"AVG_SHORT_READS_PER_REQUEST_COUNT";"MAX_SHORT_READS_PER_REQUEST_COUNT";"FULL_RETRY_READ_COUNT";"REQUESTS_WITH_FULL_RETRY_READS_COUNT";"AVG_FULL_RETRY_READS_PER_REQUEST_COUNT";"MAX_FULL_RETRY_READS_PER_REQUEST_COUNT";"FAILED_READ_COUNT"

(M_DEV_VOLUME_IO_DETAILED_STATISTICS_, 0 rows in 64us)

M_HEAP_MEMORY - (id= 48, type= public:remote:multi, reset= 0, flags= AddVolumeID AddStatisticsID Crashdump RTEdump, priv= Default)

"STATISTICS_ID";"CATEGORY";"DEPTH";"INCLUSIVE_SIZE_IN_USE [byte]";"INCLUSIVE_COUNT_IN_USE";"INCLUSIVE_ALLOCATED_SIZE [byte]";"INCLUSIVE_DEALLOCATED_SIZE [byte]";"INCLUSIVE_ALLOCATED_COUNT";"INCLUSIVE_DEALLOCATED_COUNT";"INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE [byte]";"INCLUSIVE_PEAK_ALLOCATION_SIZE [byte]";"INCLUSIVE_LIMIT [byte]";"INCLUSIVE_IN_USE_INTEGRAL";"EXCLUSIVE_SIZE_IN_USE [byte]";"EXCLUSIVE_COUNT_IN_USE";"EXCLUSIVE_ALLOCATED_SIZE [byte]";"EXCLUSIVE_DEALLOCATED_SIZE [byte]";"EXCLUSIVE_ALLOCATED_COUNT";"EXCLUSIVE_DEALLOCATED_COUNT";"EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE [byte]";"EXCLUSIVE_PEAK_ALLOCATION_SIZE [byte]";"EXCLUSIVE_ALLOC_ERRORS";"EXCLUSIVE_IN_USE_INTEGRAL";"MALLOC_PROXY_CACHE_MISSES";"FLAGS";"COMPONENT"

7;"/";0;527036910;173638;64170752045;63643715135;29942182;29768544;180711520;527140222;0;0;0;0;0;0;0;0;0;0;0;0;0;"(none)";"System"

10;"AllocateOnlyAllocator-unlimited";1;41537312;4086;88555968;47018656;55788;51702;3145728;41537368;0;0;309280;200;44432624;44123344;200;0;3145728;329480;0;0;0;"preventcheck";"System"

12;"AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>";2;40894464;13;40894464;0;13;0;3145728;40894464;0;0;0;0;0;0;0;0;0;0;0;0;0;"(none)";"System"

13;"AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks";3;40894464;13;40894464;0;13;0;3145728;40894464;0;0;40894464;13;40894464;0;13;0;3145728;40894464;0;0;0;"(none)";"System"

485;"Pool";1;440903336;169061;58228067904;57787164568;25630810;25461749;180711520;441403150;0;0;18688;155;18688;0;155;0;128;18688;0;0;0;"(none)";"System"

1665;"Pool/ncCommLibDefAllocator";2;3450544;73754;3452096;1552;73801;47;262160;3450544;0;0;3450544;73754;3452096;1552;73801;47;262160;3450544;0;0;0;"(none)";"System"

665;"Pool/TraceAllocator";2;2103968;41;17664624;15560656;97324;97283;1048576;2103968;0;0;2103968;41;17664624;15560656;97324;97283;1048576;2103968;0;0;0;"(none)";"Monitoring & Statistical Data"

498;"Pool/Statistics";2;2221880;4396;2229408184;2227186304;7323204;7318808;21504;2243552;0;0;2221880;4396;2229408184;2227186304;7323204;7318808;21504;2243552;0;0;0;"(none)";"Monitoring & Statistical Data"

504;"Pool/malloc";2;431142104;87888;53659893496;53228751392;13428060;13340172;180711520;431748422;0;0;19712;154;19712;0;154;0;128;19712;0;0;0;"(none)";"System"

2216;"Pool/malloc/hdbwebdispatcherslib.so";3;208868496;1535;44080928264;43872059768;5704783;5703248;180711520;209319904;0;0;208868496;1535;44080928264;43872059768;5704783;5703248;180711520;209319904;0;0;0;"(none)";"System"

595;"Pool/malloc/libhdbrsmemory.so";3;2097208;3;2097208;0;3;0;2097168;2097208;0;0;2097208;3;2097208;0;3;0;2097168;2097208;0;0;0;"(none)";"System"

572;"Pool/malloc/libhdbmetadatautil.so";3;29703232;54267;29703232;0;54267;0;34824;29703232;0;0;29703232;54267;29703232;0;54267;0;34824;29703232;0;0;0;"(none)";"System"

573;"Pool/malloc/libhdbmetadataobject.so";3;24418736;634;24418736;0;634;0;57352;24418736;0;0;24418736;634;24418736;0;634;0;57352;24418736;0;0;0;"(none)";"System"

574;"Pool/malloc/libhdbrskernel.so";3;71519040;1868;71542640;23600;1915;47;2097168;71519040;0;0;71519040;1868;71542640;23600;1915;47;2097168;71519040;0;0;0;"(none)";"System"

577;"Pool/malloc/libc.so.6";3;6125920;12675;8909592976;8903467056;3833749;3821074;32816;6190056;0;0;6125920;12675;8909592976;8903467056;3833749;3821074;32816;6190056;2;0;0;"(none)";"System"

586;"Pool/malloc/libhdbbasement.so";3;86104808;2787;494964536;408859728;3297904;3295117;35392000;86105448;0;0;86104808;2787;494964536;408859728;3297904;3295117;35392000;86105448;0;0;0;"(none)";"System"

499;"VirtualAlloc";1;19595264;0;5829124096;5809528832;4255062;4255062;10489856;19607552;0;0;19595264;0;5829124096;5809528832;4255062;4255062;10489856;19607552;0;0;0;"(none)";"System"

1643;"StackAllocator";1;24858624;21;24858624;0;21;0;1183744;24858624;0;0;24858624;21;24858624;0;21;0;1183744;24858624;0;0;0;"(none)";"Statement Execution & Intermediate Results"

(M_HEAP_MEMORY, 18/502 rows in 288us (313 inactive, 171 invisible rows skipped))

M_MEMORY_OBJECTS - (id= 58, type= public:remote:multi, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= Default)

"TYPE";"OBJECT_COUNT";"OBJECT_SIZE [byte]";"NON_SWAPPABLE_SIZE [byte]";"SWAPPABLE_SIZE [byte]";"PUT_COUNT";"PUT_SIZE [byte]";"EVICT_COUNT";"EVICT_SIZE [byte]";"TEMP_EVICT_COUNT";"TEMP_EVICT_SIZE [byte]";"RESIZE_COUNT";"RESIZE_DELTA_SIZE [byte]";"SHRINK_COUNT";"SHRINK_SIZE [byte]";"RETENTION_PERIOD_SHRINK_COUNT";"RETENTION_PERIOD_SHRINK_SIZE [byte]";"PAGE_LOADABLE_COLUMNS_LIMIT_SHRINK_COUNT";"PAGE_LOADABLE_COLUMNS_LIMIT_SHRINK_SIZE [byte]";"FAILED_SHRINK_COUNT";"FAILED_SHRINK_SIZE [byte]"

"/";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/GlobalSavepoint";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/Pages";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/Pages/Default";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/Pages/Static";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/Pages/Temporary";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"Persistency/Pages/RowStore";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

"SnapshotPageControlBlock";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

(M_MEMORY_OBJECTS, 9 rows in 151us)

M_MEMORY_OBJECT_FLUSH_THREAD_STATISTICS_ - (id= 56, type= private:remote, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"IS_FLUSHING";"RESOURCE_FLUSH_REQUEST_COUNT";"RESOURCE_FLUSH_EXECUTE_COUNT";"RESOURCE_FLUSH_REQUEST_SIZE [byte]";"RESOURCE_FLUSH_EXECUTE_SIZE [byte]";"LAST_PREPARE_FLUSH_DURATION [ticks]";"MAX_PREPARE_FLUSH_DURATION [ticks]";"MIN_PREPARE_FLUSH_DURATION [ticks]";"SUM_PREPARE_FLUSH_DURATION [ticks]";"AVG_PREPARE_FLUSH_DURATION [ticks]";"LAST_FLUSH_DURATION [ticks]";"MAX_FLUSH_DURATION [ticks]";"MIN_FLUSH_DURATION [ticks]";"SUM_FLUSH_DURATION [ticks]";"AVG_FLUSH_DURATION [ticks]";"LAST_PROCESSED_QUEUE_LENGTH";"MAX_PROCESSED_QUEUE_LENGTH";"MIN_PROCESSED_QUEUE_LENGTH";"SUM_PROCESSED_QUEUE_LENGTH";"AVG_PROCESSED_QUEUE_LENGTH"

0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

(M_MEMORY_OBJECT_FLUSH_THREAD_STATISTICS_, 1 rows in 11us)

M_MEMORY_RECLAIM_STATISTICS - (id= 50, type= public:remote, reset= 0, flags= AddVolumeID AddStatisticsID Crashdump RTEdump, priv= Default)

"STATISTICS_ID";"MEMORY_RECLAIM_COUNT";"LAST_MEMORY_RECLAIM_DURATION [ms]";"MAX_MEMORY_RECLAIM_DURATION [ms]";"MIN_MEMORY_RECLAIM_DURATION [ms]";"SUM_MEMORY_RECLAIM_DURATION [ms]";"AVG_MEMORY_RECLAIM_DURATION [ms]";"LAST_SYNCHRONIZATION_DURATION [ms]";"MAX_SYNCHRONIZATION_DURATION [ms]";"MIN_SYNCHRONIZATION_DURATION [ms]";"SUM_SYNCHRONIZATION_DURATION [ms]";"AVG_SYNCHRONIZATION_DURATION [ms]";"LAST_DEFRAGMENTATION_DURATION [ms]";"MAX_DEFRAGMENTATION_DURATION [ms]";"MIN_DEFRAGMENTATION_DURATION [ms]";"SUM_DEFRAGMENTATION_DURATION [ms]";"AVG_DEFRAGMENTATION_DURATION [ms]";"LAST_DEFRAGMENTATION_SIZE [byte]";"MAX_DEFRAGMENTATION_SIZE [byte]";"MIN_DEFRAGMENTATION_SIZE [byte]";"SUM_DEFRAGMENTATION_SIZE [byte]";"AVG_DEFRAGMENTATION_SIZE [byte]";"LAST_MEMORY_OBJECT_UNLOAD_DURATION [ms]";"MAX_MEMORY_OBJECT_UNLOAD_DURATION [ms]";"MIN_MEMORY_OBJECT_UNLOAD_DURATION [ms]";"SUM_MEMORY_OBJECT_UNLOAD_DURATION [ms]";"AVG_MEMORY_OBJECT_UNLOAD_DURATION [ms]";"LAST_MEMORY_OBJECT_UNLOAD_SIZE [byte]";"MAX_MEMORY_OBJECT_UNLOAD_SIZE [byte]";"MIN_MEMORY_OBJECT_UNLOAD_SIZE [byte]";"SUM_MEMORY_OBJECT_UNLOAD_SIZE [byte]";"AVG_MEMORY_OBJECT_UNLOAD_SIZE [byte]"

486;154580;3460;41904;0;985943;6;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;1552134144;0;12956733440;83818

(M_MEMORY_RECLAIM_STATISTICS, 1 rows in 27us)

MEMORY_RECLAIM_TASKS - (id= -3, type= none:<invalid>, reset= 0, flags= AddVolumeID Crashdump RTEdump, priv= <none>)

"TASK_NAME";"TOTAL_NUM_BATCHES";"PEAK_RUN_NUM_BATCHES";"TOTAL_ELAPSED_TIME [ms]";"PEAK_RUN_ELAPSED_TIME [ms]";"TOTAL_BATCH_FINISH_WAIT_TIME [ms]";"PEAK_RUN_BATCH_FINISH_WAIT_TIME [ms]";"TOTAL_NUM_TASKS";"PEAK_RUN_NUM_TASKS";"TOTAL_ELAPSED_TASK_TIME [ms]";"PEAK_RUN_ELAPSED_TASK_TIME [ms]";"TOTAL_TASK_WAIT_TIME [ms]";"PEAK_RUN_TASK_WAIT_TIME [ms]";"TOTAL_NUM_FAILED_TASK_REGISTRATIONS";"PEAK_RUN_NUM_FAILED_TASK_REGISTRATIONS";"TOTAL_NUM_SERIALIZED_TASKS";"PEAK_RUN_NUM_SERIALIZED_TASKS";"TOTAL_ELAPSED_TIME_FOR_SERIALIZED_TASKS [ms]";"PEAK_RUN_ELAPSED_TIME_FOR_SERIALIZED_TASKS [ms]";"ITEM_NAME";"TOTAL_NUM_ITEMS";"PEAK_RUN_NUM_ITEMS";"TOTAL_SIZE_ITEMS [byte]";"PEAK_RUN_SIZE_ITEMS [byte]";"DEAIL_TIME_TYPE";"TOTAL_DETAIL_TIME [ms]";"PEAK_RUN_DETAIL_TIME [ms]";"ALLOW_PARALLELIZATION"

"IPMM/Defragmentation";41065;5;150619;4;0;0;41065;5;150586;4;150220;4;0;0;41065;5;150586;4;"num wait repetitions";41065;5;0;0;"";0;0;0

"IPMM/ForcedCompaction";45;1;3672;857;0;0;45;1;3672;857;3671;857;0;0;45;1;3672;857;"num wait repetitions";45;1;0;0;"";0;0;0

"PoolAllocator/ReturnFreelistToOS";49362;0;222;0;0;0;49362;0;201;0;0;0;0;0;49362;0;201;0;"freed segments";12;0;3221225472;0;"";0;0;0

"PoolAllocator/CollectGarbage";39824;0;15137;0;68;0;358416;0;15759;0;708;0;0;0;339872;0;14915;0;"freed memory parts";14;0;1683177472;0;"deallocate/unmap";3;0;1

"PoolAllocator/FreeListOfSystemPagesWithFailedFree";9287;0;5;0;0;0;9287;0;3;0;0;0;0;0;9287;0;3;0;"freed segments";0;0;0;0;"";0;0;0

"ResMan/shrink/RebuildHeap/ClearHeap";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"heap resources";0;0;0;0;"";0;0;1

"ResMan/shrink/RebuildHeap/BuildHeap";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"heap resources";0;0;0;0;"";0;0;1

"ResMan/shrink/RebuildHeap/ClearLeftChild";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"free resources";0;0;0;0;"";0;0;1

"ResMan/shrink/RebuildHeap/GetFreeResources";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"traversed resources";0;0;0;0;"";0;0;1

"ResMan/shrink/GetLock";0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"";0;0;0;0;"";0;0;0

(MEMORY_RECLAIM_TASKS, 10 rows in 56us)

M_ACTIVE_STATEMENTS_ - (id= 166, type= private:local, reset= 0, flags= <none>, priv= <none>)

"CONNECTION_ID";"STATEMENT_ID";"STATEMENT_HASH";"OBJECT_HANDLE";"ALLOCATED_NUMA_NODE";"PLAN_ID";"START_MVCC_TIMESTAMP";"SNAPSHOT_TIMESTAMP";"COMPILED_TIME";"LAST_EXECUTED_TIME";"LAST_ACTION_TIME";"STATEMENT_STATUS";"STATEMENT_INTERNAL_STATUS";"CURRENT_THREAD_ID";"STATEMENT_STRING";"ALLOCATED_MEMORY_SIZE [byte]";"USED_MEMORY_SIZE [byte]";"RECOMPILE_COUNT";"RETRY_COUNT";"RETRY_COUNT_BY_SERIALIZATION";"RETRY_COUNT_BY_SERIALIZATION_WITH_TIMEOUT";"EXECUTION_COUNT";"EXECUTION_COUNT_BY_ROUTING_COUNT";"EXECUTION_COUNT_BY_UPDATE_TRANSACTION";"EXECUTION_COUNT_BY_DIRECT_RECORD_PASSING";"EXECUTION_COUNT_BY_JOBEXECUTOR";"AVG_EXECUTION_TIME";"MAX_EXECUTION_TIME";"MIN_EXECUTION_TIME";"TOTAL_EXECUTION_TIME";"AVG_CURSOR_DURATION";"MAX_CURSOR_DURATION";"MIN_CURSOR_DURATION";"TOTAL_CURSOR_DURATION";"AVG_EXECUTION_MEMORY_SIZE";"MAX_EXECUTION_MEMORY_SIZE";"MIN_EXECUTION_MEMORY_SIZE";"TOTAL_EXECUTION_MEMORY_SIZE";"AVG_LOCKWAIT_TIME";"MAX_LOCKWAIT_TIME";"MIN_LOCKWAIT_TIME";"TOTAL_LOCKWAIT_TIME";"TOTAL_LOCKWAIT_COUNT";"AVG_PREPARATION_TIME";"MAX_PREPARATION_TIME";"MIN_PREPARATION_TIME";"TOTAL_PREPARATION_TIME";"TOTAL_PREPARATION_COUNT";"HAS_HOLDABLE_CURSOR";"CURSOR_TYPE";"CACHED";"IN_USE";"PARENT_STATEMENT_ID";"APPLICATION_SOURCE";"ISOLATION_LEVEL";"TOTAL_SAMPLED_COUNT";"AVG_SAMPLED_CPU_TIME";"MAX_SAMPLED_CPU_TIME";"MIN_SAMPLED_CPU_TIME";"TOTAL_SAMPLED_CPU_TIME";"AVG_SAMPLED_WAIT_TIME";"MAX_SAMPLED_WAIT_TIME";"MIN_SAMPLED_WAIT_TIME";"TOTAL_SAMPLED_WAIT_TIME";"AVG_CHILD_THREAD_COUNT_DURING_PREPARATION";"TOTAL_CHILD_THREAD_COUNT_DURING_PREPARATION";"AVG_CHILD_THREAD_COUNT_DURING_EXECUTION";"TOTAL_CHILD_THREAD_COUNT_DURING_EXECUTION";"AVG_CHILD_THREAD_COUNT_DURING_FETCH";"TOTAL_CHILD_THREAD_COUNT_DURING_FETCH";"AVG_CHILD_THREAD_COUNT_DURING_CLOSE";"TOTAL_CHILD_THREAD_COUNT_DURING_CLOSE";"AVG_NETWORK_REQUEST_COUNT_DURING_PREPARATION";"TOTAL_NETWORK_REQUEST_COUNT_DURING_PREPARATION";"AVG_NETWORK_REQUEST_COUNT_DURING_EXECUTION";"TOTAL_NETWORK_REQUEST_COUNT_DURING_EXECUTION";"AVG_NETWORK_REQUEST_COUNT_DURING_FETCH";"TOTAL_NETWORK_REQUEST_COUNT_DURING_FETCH";"AVG_NETWORK_REQUEST_COUNT_DURING_CLOSE";"TOTAL_NETWORK_REQUEST_COUNT_DURING_CLOSE";"PRIORITY";"STATEMENT_MEMORY_LIMIT";"STATEMENT_THREAD_LIMIT";"WORKLOAD_CLASS_NAME"

(M_ACTIVE_STATEMENTS_, 0 rows in 98us)

Some information about current context:

TID: -1, UTID: -1, CID: -1, LCID: 0, parent: 1289, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "Search for trace files to compress", detail: "", command: ""Current allocated bytes (for this process)= 1359212544.

[OK]

--

[DONE] (2016-04-25 16:17:34 088 Local)

[OK]

--

Hi Mayank,

Yes you got a response from the ticket as far as I can see. Other than that SAP provides means to expedite the message processing. Talk to your account executive about those. SCN discussion is *not* one of them.

Please double check the answer and keep the discussion there. Share the results here afterwards.

BRs,

Lucas de Oliveira

Hi Sakthi,

Are you able to connect to your HANA system HANA Studio?

If yes you should be able to see the version on the first screen called OVERVIEW for the system

Sunil

Hello,

Unfortunately this is not helpful. (The webdispatcher OOM do not tell us much)

I think it would be best if you open a ticket for this.

Please send me the incident number when you have created it.

KR

Michael 

Hi Michael, thanks for your support.

The other OOM Trace files aviable are from nameserver and preprocessor

On both seems to have the same error failing to allocate xxxxx bytes

[MEMORY_OOM]  Information about current out of memory situation: (2016-04-26 17:51:34 328 Local)

OUT OF MEMORY occurred.

Host: uyclsxxx.mdp.local

Executable: hdbnameserver

PID: 628

Failed to allocate 55544 byte.

Current callstack:

1: 0x00007fc5bea5e88c in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, unsigned long, bool)+0xb8 at PoolAllocator.cpp:777 (libhdbbasis.so)

2: 0x00007fc5bea5f5c0 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x250 at PoolAllocator.cpp:1449 (libhdbbasis.so)

3: 0x00007fc5d756b8de in ltt::allocator::allocate(unsigned long)+0x2a at memory.cpp:83 (libhdbbasement.so)

4: 0x00007fc5d7260ebf in ltt::vector<TrexThreads::ThreadInfo>::reserve_(unsigned long)+0x12b at memory.hpp:845 (libhdbbasement.so)

5: 0x00007fc5d725bba0 in TrexThreads::ThreadManager::getStatistics(ltt_adp::vector<TrexThreads::ThreadInfo, ltt::integral_constant<bool, true> >*, bool, bool, bool, ltt_adp::map<unsigned long, ltt_adp::vector<unsigned long, ltt::integral_constant<bool, true> >, ltt::less<unsigned long>, ltt::integral_constant<bool, true> >*)+0xdd0 at vector.hpp:1143 (libhdbbasement.so)

6: 0x00007fc5c1c86dfc in ptime::traceThreadLockInfo()+0x138 at ServiceThreadSamplesMonitor.cc:196 (libhdbrskernel.so)

7: 0x00007fc5c1c8be65 in ptime::ServiceThreadSamplerThread::run(void*)+0x11 at ServiceThreadSamplesMonitor.cc:159 (libhdbrskernel.so)

8: 0x00007fc5d7252c80 in TrexThreads::PoolThread::run()+0xb10 at PoolThread.cpp:294 (libhdbbasement.so)

9: 0x00007fc5d7254e20 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:136 (libhdbbasement.so)

10: 0x00007fc5be8dbf29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

11: 0x00007fc5be8dcdfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

Memory consumption information of last failing ProvideMemory, PM-INX=1527655:

IPMM short info:

GLOBAL_ALLOCATION_LIMIT (GAL) = 138.67gb (148899934208b), SHARED_MEMORY = 21.69gb (23298207744b), CODE_SIZE = 1.62gb (1742049280b), OOM_RESERVATION_ALLOCATOR_SIZE = 288.54mb (302563328b)

PID=628 (hdbnameserver), PAL=138.79gb (149034151936b), AB=7.92gb (8505503744b), UA=0, U=5.86gb (6296738381b), FSL=0

PID=966 (hdbcompileserve), PAL=138.79gb (149034151936b), AB=1.06gb (1143971840b), UA=0, U=300.78mb (315401201b), FSL=0, Blocked=2.20gb (2363805696b)

PID=968 (hdbpreprocessor), PAL=138.79gb (149034151936b), AB=988.55mb (1036574720b), UA=0, U=217.32mb (227880229b), FSL=0

PID=995 (hdbindexserver), PAL=138.79gb (149034151936b), AB=32.38gb (34772930560b), UA=0, U=28.70gb (30824124089b), FSL=0

PID=1272 (hdbwebdispatche), PAL=138.79gb (149034151936b), AB=1.26gb (1358028800b), UA=0, U=501.48mb (525845838b), FSL=0

PID=7008 (hdbindexserver), PAL=138.79gb (149034151936b), AB=70.45gb (75654856704b), UA=0, U=65.49gb (70328167436b), FSL=0

Total allocated memory= 137.38gb (147512123392b)

Total used memory     = 124.38gb (133558414198b)

Sum AB                = 122471866368

Sum Used              = 108518157174

Heap memory fragmentation: 9

Top allocators (component, name, size). Ordered descending by inclusive_size_in_use.

1: System:                                     /                                                                       5.86gb (6296738381b)

2: System:                                     Pool                                                                    4.85gb (5208601677b)

3: System:                                     Pool/malloc                                                             3.25gb (3492119078b)

4: System:                                     Pool/malloc/libhdbrskernel.so                                           2.62gb (2824337496b)

5: System:                                     AllocateOnlyAllocator-unlimited                                         909.67mb (953866528b)

6: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 894mb (937426944b)

7: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>                       894mb (937426944b)

8: System:                                     Pool/RowEngine                                                          411.04mb (431007360b)

9: Column Store Tables:                        Pool/SerializedObject                                                   355.15mb (372405775b)

10: Caches:                                     Pool/RowEngine/SQLPlan                                                  236.02mb (247494336b)

11: System:                                     Pool/malloc/libhdbmetadataobject.so                                     166.41mb (174503408b)

12: Monitoring & Statistical Data:              Pool/Statistics                                                         133.25mb (139728424b)

13: System:                                     Pool/Text                                                               128.51mb (134756124b)

14: System:                                     Pool/Text/AEText                                                        128.50mb (134752332b)

15: Column Store Tables:                        Pool/ColumnStoreTables                                                  126.68mb (132837383b)

16: System:                                     Pool/PersistenceManager                                                 126.35mb (132488752b)

17: Statement Execution & Intermediate Results: StackAllocator                                                          123.33mb (129323008b)

18: Statement Execution & Intermediate Results: Pool/TransactionManager                                                 115.79mb (121425040b)

19: System:                                     Pool/Text/AEText/split_positional_index                                 97.02mb (101742024b)

20: Statement Execution & Intermediate Results: Pool/malloc/libhdbcsapi.so                                              95.79mb (100453294b)

Top allocators (component, name, size). Ordered descending by exclusive_size_in_use.

1: System:                                     Pool/malloc/libhdbrskernel.so                                                      2.62gb (2824337496b)

2: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks            894mb (937426944b)

3: Column Store Tables:                        Pool/SerializedObject                                                              355.15mb (372405775b)

4: Caches:                                     Pool/RowEngine/SQLPlan                                                             236.02mb (247494336b)

5: System:                                     Pool/malloc/libhdbmetadataobject.so                                                166.41mb (174503408b)

6: Monitoring & Statistical Data:              Pool/Statistics                                                                    133.25mb (139728424b)

7: Statement Execution & Intermediate Results: StackAllocator                                                                     123.33mb (129323008b)

8: Statement Execution & Intermediate Results: Pool/TransactionManager                                                            115.79mb (121425040b)

9: System:                                     Pool/Text/AEText/split_positional_index                                            97.02mb (101742024b)

10: Statement Execution & Intermediate Results: Pool/malloc/libhdbcsapi.so                                                         95.79mb (100453294b)

11: Monitoring & Statistical Data:              Pool/RowEngine/MonitorView/Resident/M_DEV_TRANSACTIONS_HISTORY                     87.73mb (92000016b)

12: System:                                     Pool/malloc/libhdbexpression.so                                                    52.62mb (55180072b)

13: System:                                     Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page                         52.59mb (55148544b)

14: System:                                     Pool/malloc/libhdbns.so                                                            49.37mb (51771224b)

15: Monitoring & Statistical Data:              Pool/MergeMonitor                                                                  48.82mb (51200512b)

16: System:                                     Pool/malloc/libhdbcalcengine.so                                                    43.25mb (45356336b)

17: Column Store Tables:                        Pool/ColumnStoreTables/Main/Dictionary/RoDict                                      42.33mb (44396471b)

18: System:                                     Pool/malloc/libhdbbasement.so                                                      42.22mb (44278672b)

19: System:                                     Pool/ResourceContainer                                                             32.20mb (33771048b)

20: System:                                     Pool/FileAllocator/FileFactory[/usr/sap/HED/SYS/global/hdb/log/mnt00001/hdb00001/] 30mb (31461840b)

Out of memory reasons:

!:! <1> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <2> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <3> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <4> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <5> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <6> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <7> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <8> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <9> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <10> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <11> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <12> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <13> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <14> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <15> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <16> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <17> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <18> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <19> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <20> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <21> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <22> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <23> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <24> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <25> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <26> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <27> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <28> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <29> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <30> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <31> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <32> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <33> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <34> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <35> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <36> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <37> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <38> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <39> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <40> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <41> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <42> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <43> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <44> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <45> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <46> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <47> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <48> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <49> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <50> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

At most one OOM dump is written in approximately 86400s.

Raw memory information (2):

Allocated=8864755712 (accounted=8864755712, unaccounted=0)

Reserved size for libraries (data segments only)=852029440 (data segments size (libraries)=852029440)

Raw Pool Allocator size=6588932096 (including huge alignment raw size=2965504, V=3670016)

Allocate Only size=1296207872(limited=16777216, unlimited=1279430656 [including emergency allocators], unreserved Allocate Only size=0)Raw Limited Allocate Only Allocator size=16777216

Stack size=122220544

Free list size=0

Memory failure free list size=0

Big StackReservationScope reservations=0 (limited=0, unlimited=0)

VirtualAlloc/MMap = 4587520

JumpTable = 778240

StatAllocOwnBlocks =524288

OOMReservationAlloc (not part of 'allocated'!) = 0

(System memory: 10535109345280 - 10527095390208 = 8013955072, + data sagements=8865984512)

pureVirtualHugeAlignmentSize=704512

Looks fine!

Compaction tasks

Last compaction: 453. Reason: IPMM (other process). Request ID=215003.

ReserveRequest=53.53mb (56131584b), ShrinkRequest=4194303.99tb (4611686018427383808b)

Compaction tasks for compaction 453:

  Limits:  TempLimit=8.25gb (8864718848b), ProcessLimit=138.79gb (149034151936b).

  Alloc status: Used=6.33gb (6804412431b), Allocated=8.25gb (8864718848b).

  Fragmentation: 24%.

  Freelist: Size=0b.

* Forced garbage collection (fragmentation too big).

  Alloc status: Used=6.28gb (6748280847b), Allocated=8.30gb (8920850432b).

  Fragmentation: 25%.

  Freelist: Size=0b.

  Compactor status before compaction: Size=697.45mb (731334031b), Freeable=59.16mb (62037885b).

* Compact: reserve=53.53mb (56131584b), shrink=4194303.99tb (4611686018427383808b).

  Compactor status after compaction: Size=697.45mb (731334031b), Freeable=59.16mb (62037885b).

  Alloc status: Used=6.28gb (6748280847b), Allocated=8.30gb (8920850432b).

  Fragmentation: 25%.

  Freelist: Size=0b.

* Weak garbage collection.

* Forced garbage collection..

  Alloc status: Used=6.28gb (6748280847b), Allocated=8.30gb (8920850432b).

  Fragmentation: 25%.

  Freelist: Size=0b.

Result: Reserved=0b.

Called with parameter 'forced garbage collection'.

Called with parameter 'forced' (wait for lock).

Inter Process Memory Management

IPMM shared memory segment: SAP_IPMM_0028_HED

SId=        Process secondary identifier

AB=         Allocated bytes by allocators for heap and stacks (inclusive allocator cache)

UA=         Unaccounted memory (by unlimited allocators for exceptions etc.), maybe overcommit

U=          Used bytes for heap and stacks (without allocator cache)

FSL=        Free segment list size

SMAS/SMDS=  System memory traffic information (accumulation of allocations/deallocations)

PAL=        Process allocation limit

TPAL=       Temp process allocation limit

Blocked=    Memory (TPAL!) blocked/reserved for this process

W=          Weight of process

[0] PID=628 (state=R), SId=300674672, compactors active, alive, process name: hdbnameserver

    AB=8.25gb (8864755712b), UA=0b, U=6.33gb (6804595191b), FSL=0b, SMAS=9.58tb (10535109345280b), SMDS=9.57tb (10527095390208b), PAL=138.79gb (149034151936b), TPAL=8.33gb (8953409536b), W=100

[1] PID=966 (state=S), SId=300674673, compactors active, alive, process name: hdbcompileserve

    AB=1.08gb (1161891840b), UA=0b, U=312.79mb (327986769b), FSL=0b, SMAS=34.04gb (36552007680b), SMDS=33.65gb (36139896832b), PAL=138.79gb (149034151936b), TPAL=2.20gb (2363437056b), Blocked=2.20gb (2363805696b), W=100

[2] PID=968 (state=S), SId=300674674, compactors active, alive, process name: hdbpreprocessor

    AB=1.21gb (1305010176b), UA=0b, U=220.32mb (231025893b), FSL=0b, SMAS=11.13tb (12244274610176b), SMDS=11.13tb (12243727507456b), PAL=138.79gb (149034151936b), TPAL=1.21gb (1305010176b), W=100

[3] PID=995 (state=S), SId=300674675, compactors active, alive, process name: hdbindexserver

    AB=31.99gb (34350096384b), UA=0b, U=27.90gb (29964866354b), FSL=0b, SMAS=7.86tb (8652529795072b), SMDS=7.83tb (8619030663168b), PAL=138.79gb (149034151936b), TPAL=31.99gb (34350096384b), W=100

[4] PID=1272 (state=S), SId=300674676, compactors active, alive, process name: hdbwebdispatche

    AB=1.51gb (1627648000b), UA=0b, U=502.66mb (527084998b), FSL=0b, SMAS=14.23tb (15651698860032b), SMDS=14.23tb (15650849234944b), PAL=138.79gb (149034151936b), TPAL=1.52gb (1641381888b), W=100

[5] PID=7008 (state=S), SId=300674677, compactors active, alive, process name: hdbindexserver

    AB=69.14gb (74245894144b), UA=0b, U=62.29gb (66885101749b), FSL=0b, SMAS=11.77tb (12945721135104b), SMDS=11.70tb (12872594083840b), PAL=138.79gb (149034151936b), TPAL=69.92gb (75077996544b), W=100

GLOBAL_MAX_ALLOCATION_LIMIT=138.79gb (149034151936b)

SHARED_MEMORY=21.69gb (23298207744b)

OOM_RESERVATION_ALLOCATORS=288.54mb (302563328b)

CODE_SIZE=1.62gb (1742049280b)

Sum TPAL (including shm, code size and IPMM reservation and IPMM OOM reservation)=138.79gb (149034151936b) (cached sum=138.79gb (149034151936b)), IPMM reservation=0b, emergency reservation=128mb (134217728b)

#processes=6, weight sum=600, cached weight sum=600, #checks for cached allocation limit sum=11148944, #terminated processes since last reset=0

Early exits are currently allowed. Dumping of IPMM flight recorder is currently enabled.

109 early exits since creation of IPMM, reclaim-memory-counter=1788120. Reclaim memory 1788093 failed, only 212303872b were free.

Last reclaim memory started to wait for IPMM lock since 2016-04-26 17:51:33.925.

Last reclaim memory got IPMM lock at 2016-04-26 17:51:34.340.

IPMM flight recorder is not verbose.

User ids respected in shm calculation: [1000]

Parallel memory reclaim tasks are activated.

This is an opt build.

Peak used memory: 125.63gb (134897761776b) with corresponding values (Heap=102.31gb (109857504752b), Shared=21.69gb (23298207744b), Code=1.62gb (1742049280b)).

Peak heap memory: 102.31gb (109857504752b). Peak shared memory: 21.69gb (23298207744b). Peak code size: 1.62gb (1742049280b)

=========================================

Runtime dump protocol for PID 968

Rundirectory: /usr/sap/HED/HDB10/uyclsxxx.mdp.local

Process start time: 2016-04-14 10:21:56 973 Local

Runtime dump time: 2016-04-25 15:08:18 853 Local

SectionTimeout: 30sec

KillTimeout: 300sec

Runtime Dump Filter: (OOM Runtime dump filter)

=========================================

Table of contents:

   [BUILD]  Build information

   [STACK_SHORT]  Short call stacks and pending exceptions of all threads

   [PROCESS_INFO]  Process Info

   [OS_MEMORY]  Operating system information about memory

   [THREADS]  Running threads

   [MEMORY_OOM]  Information about current out of memory situation

[BUILD]  Build information: (2016-04-25 15:08:18 853 Local)

Version       : 1.00.102.01.1444147999 (fa/newdb100_rel)

Build host    : ld7272

Build time    : 2015-10-06 19:16:10

Platform      : linuxx86_64

Compiler      : gcc (SAP release 20150922, based on SUSE gcc47-4.7.2_20130108-0.15.45.6763.0.PTF.871552) 4.7.2 20130108 [gcc-4_7-branch revision 195014]

Branch        : fa/newdb100_rel

Git hash      : 319000a825a2a9e19fd235302bd80b9a4ff8bb62

Git mergetime : 2015-10-06 18:13:19

Weekstone     : 0000.00.0

[OK]

--

[STACK_SHORT]  Short call stacks and pending exceptions of all threads: (2016-04-25 15:08:18 853 Local)

2324[thr=1062]: JobWrk0003 is inactive

--

2582[thr=1321]: JobWrk0004 is inactive

--

2748[thr=1454]: Pool is inactive

--

54292[thr=13394]: JobWrk0005 is inactive

--

522995[thr=28979]: Pool is inactive

--

879757[thr=16378]: Pool is inactive

--

2216776[thr=23586]: Pool is inactive

--

5[thr=0]: <SYSTEM> is inactive

--

691[thr=980]: TimerThread is inactive

--

720[thr=983]: TraceSegmentCompressor at

1: 0x00007f08ee6614e2 in Execution::ContextFunctions::printStack(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool)+0xc0 at ContextFunctions.cpp:628 (libhdbbasis.so)

2: 0x00007f08ee51c15e in Diagnose::RuntimeDumpSectionStackShort::dumpContext(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::RuntimeDumpTimer&, Diagnose::FaultDescriptor*)+0xfa at CrashDump.cpp:959 (libhdbbasis.so)

3: 0x00007f08ee51eb38 in Diagnose::RuntimeDumpSectionStackShort::dump(ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::RuntimeDumpTimer&, Diagnose::FaultDescriptor*)+0x784 at CrashDump.cpp:982 (libhdbbasis.so)

4: 0x00007f08ee51ba44 in Diagnose::RuntimeDumpHelper::writeSectionNoSEH(Diagnose::RuntimeDumpSectionBase&, Diagnose::InternalRuntimeDumpTimer&)+0x180 at CrashDump.cpp:534 (libhdbbasis.so)

5: 0x00007f08ee516a62 in Diagnose::CrashDump::executeInternal(ltt::basic_ostream<char, ltt::char_traits<char> >&, Diagnose::FaultDescriptor*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*, char const*)+0x690 at CrashDump.cpp:1928 (libhdbbasis.so)

6: 0x00007f08ee5180aa in Diagnose::CrashDump::executeWithFilename(char const*, Diagnose::FaultDescriptor*, char const*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*)+0xd6 at CrashDump.cpp:1662 (libhdbbasis.so)

7: 0x00007f08ee51856a in Diagnose::RuntimeDump::createSystemDump(char const*, Diagnose::LocalRuntimeDumpSection**, unsigned long, Diagnose::RuntimeDumpFilter const*)+0x46 at CrashDump.cpp:2325 (libhdbbasis.so)

8: 0x00007f08ee859214 in MemoryManager::reportOOM(unsigned long)+0x2d0 at ReportMemoryProblems.cpp:1063 (libhdbbasis.so)

9: 0x00007f08ee86c88c in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, unsigned long, bool)+0xb8 at PoolAllocator.cpp:777 (libhdbbasis.so)

10: 0x00007f08ee86d5c0 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x250 at PoolAllocator.cpp:1449 (libhdbbasis.so)

11: 0x00007f08eae2e651 in __alloc_dir+0x60 (libc.so.6)

12: 0x00007f08ee9382b4 in System::UX::opendir(char const*)+0x20 at SystemCallsUNIX.cpp:75 (libhdbbasis.so)

13: 0x00007f08ee745de8 in FileAccess::DirectoryEntry::findFirst()+0x14 at SimpleFile.cpp:600 (libhdbbasis.so)

14: 0x00007f08ee74bd9b in FileAccess::DirectoryEntry::DirectoryEntry(char const*)+0x467 at SimpleFile.cpp:99 (libhdbbasis.so)

15: 0x00007f08ee622ec0 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x250 at TraceSegment.cpp:158 (libhdbbasis.so)

16: 0x00007f08ee6e9f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

17: 0x00007f08ee6eadfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

--

2118[thr=979]: MemoryCompactor is inactive

--

2123[thr=981]: NetworkChannelCompletionThread is inactive

--

2124[thr=982]: NetworkChannelCompletionThread is inactive

--

2140[thr=984]: WakeupThread is inactive

--

2236[thr=985]: SignalQueue is inactive

--

2259[thr=988]: PreprocessorUtil at

1: 0x00007f08eb127aa1 in sem_timedwait+0x60 (libpthread.so.0)

2: 0x00007f08ee8df45b in Synchronization::InterProcessMutex::waitEvent(void*, void*)+0x37 at InterProcessMutex.cpp:346 (libhdbbasis.so)

3: 0x00007f08ee8df723 in Synchronization::InterProcessMutex::timedLock(void*, unsigned long)+0x1a0 at InterProcessMutex.cpp:315 (libhdbbasis.so)

4: 0x00007f08ee808abc in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLock(bool, unsigned long, unsigned long, bool&)+0x48 at GlobalMemoryHandler.cpp:6830 (libhdbbasis.so)

5: 0x00007f08ee808d8c in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemoryHlp(MemoryManager::IpmmMasterScope&, unsigned long)+0x88 at GlobalMemoryHandler.cpp:6804 (libhdbbasis.so)

6: 0x00007f08ee808eb8 in MemoryManager::GlobalMemoryHandler::pmAcquireIPMMLockOrSupportProvideMemory(MemoryManager::IpmmMasterScope&, ltt::auto_object<Synchronization::LockedScope<Synchronization::NamedProcessMutexNoAlloc, true> >&, unsigned long, bool)+0xe4 at GlobalMemoryHandler.cpp:6762 (libhdbbasis.so)

7: 0x00007f08ee81815c in MemoryManager::GlobalMemoryHandler::provideMemoryAndReturnSize(bool, bool, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x698 at GlobalMemoryHandler.cpp:4603 (libhdbbasis.so)

8: 0x00007f08ee818cd5 in MemoryManager::GlobalMemoryHandler::provideMemory(unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x41 at GlobalMemoryHandler.cpp:4387 (libhdbbasis.so)

9: 0x00007f08ee7b7a45 in MemoryManager::AllocatorConfigurationSPI::provideMemory(unsigned long, unsigned long, bool, char const*, unsigned long)+0x21 at AllocatorConfigurationImpl.cpp:2765 (libhdbbasis.so)

10: 0x00007f08ee8840d1 in MemoryManager::VirtualAllocTracer::reserveMemory(unsigned long&, bool, bool, void*, unsigned long)+0x290 at VirtualAllocTracer.cpp:187 (libhdbbasis.so)

11: 0x00007f08ee825e6c in MemoryManager::mmapOverride(void*, unsigned long, int, int, int, long, void*)+0x2b8 at MallocProxy.cpp:1660 (libhdbbasis.so)

12: 0x00007f08eadec2ec in _IO_file_doallocate_internal+0x78 (libc.so.6)

13: 0x00007f08eadf97ac in _IO_doallocbuf_internal+0x28 (libc.so.6)

14: 0x00007f08eadf7d64 in _IO_new_file_underflow+0x1b0 (libc.so.6)

15: 0x00007f08eadf97ee in _IO_default_uflow_internal+0xa (libc.so.6)

16: 0x00007f08eade0ca0 in _IO_vfscanf_internal+0x6060 (libc.so.6)

17: 0x00007f08eade95e8 in __fscanf+0x84 (libc.so.6)

18: 0x00007f08f49bfb6d in get_loadavg+0x49 at perfmon.c:519 (libhdbbasement.so)

19: 0x00007f08f61fc59a in preprocessor::PreprocessorUtil::run(void*&)+0x296 at PreprocessorPool.cpp:630 (hdbpreprocessor)

20: 0x00007f08ee6e9f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

21: 0x00007f08ee6eadfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

--

2261[thr=986]: PreprocessorPool is inactive

--

2264[thr=987]: SelfWatchDog is inactive

--

2275[thr=1032]: Pool is inactive

--

2316[thr=1058]: JobexWatchdog is inactive

--

2318[thr=1059]: JobexMainDispatcher is inactive

--

2320[thr=1060]: JobWrkDiag is inactive

--

2322[thr=1061]: JobWrk0002 is inactive

--

[OK]

--

[PROCESS_INFO]  Process Info: (2016-04-25 15:08:18 872 Local)

system pagesize: 4096

dumping process info, all memory sizes in system pages!

   PID       PPID        UID       VIRT        RES        SHR        TRS        DRS        LRS COMMAND

     1          0          0       4841        386        306         35          0        105 /sbin/init

     2          0          0          0          0          0          0          0          0 [kthreadd]

     3          2          0          0          0          0          0          0          0 [migration/0]

     4          2          0          0          0          0          0          0          0 [ksoftirqd/0]

     5          2          0          0          0          0          0          0          0 [migration/0]

     6          2          0          0          0          0          0          0          0 [watchdog/0]

     7          2          0          0          0          0          0          0          0 [migration/1]

     8          2          0          0          0          0          0          0          0 [migration/1]

     9          2          0          0          0          0          0          0          0 [ksoftirqd/1]

    10          2          0          0          0          0          0          0          0 [watchdog/1]

    11          2          0          0          0          0          0          0          0 [migration/2]

    12          2          0          0          0          0          0          0          0 [migration/2]

    13          2          0          0          0          0          0          0          0 [ksoftirqd/2]

    14          2          0          0          0          0          0          0          0 [watchdog/2]

    15          2          0          0          0          0          0          0          0 [migration/3]

    16          2          0          0          0          0          0          0          0 [migration/3]

    17          2          0          0          0          0          0          0          0 [ksoftirqd/3]

    18          2          0          0          0          0          0          0          0 [watchdog/3]

    19          2          0          0          0          0          0          0          0 [migration/4]

    20          2          0          0          0          0          0          0          0 [migration/4]

    21          2          0          0          0          0          0          0          0 [ksoftirqd/4]

    22          2          0          0          0          0          0          0          0 [watchdog/4]

    23          2          0          0          0          0          0          0          0 [migration/5]

    24          2          0          0          0          0          0          0          0 [migration/5]

    25          2          0          0          0          0          0          0          0 [ksoftirqd/5]

    26          2          0          0          0          0          0          0          0 [watchdog/5]

    27          2          0          0          0          0          0          0          0 [migration/6]

    28          2          0          0          0          0          0          0          0 [migration/6]

    29          2          0          0          0          0          0          0          0 [ksoftirqd/6]

    30          2          0          0          0          0          0          0          0 [watchdog/6]

    31          2          0          0          0          0          0          0          0 [migration/7]

    32          2          0          0          0          0          0          0          0 [migration/7]

    33          2          0          0          0          0          0          0          0 [ksoftirqd/7]

    34          2          0          0          0          0          0          0          0 [watchdog/7]

    35          2          0          0          0          0          0          0          0 [events/0]

    36          2          0          0          0          0          0          0          0 [events/1]

    37          2          0          0          0          0          0          0          0 [events/2]

    38          2          0          0          0          0          0          0          0 [events/3]

    39          2          0          0          0          0          0          0          0 [events/4]

    40          2          0          0          0          0          0          0          0 [events/5]

    41          2          0          0          0          0          0          0          0 [events/6]

    42          2          0          0          0          0          0          0          0 [events/7]

    43          2          0          0          0          0          0          0          0 [cgroup]

    44          2          0          0          0          0          0          0          0 [khelper]

    45          2          0          0          0          0          0          0          0 [netns]

    46          2          0          0          0          0          0          0          0 [async/mgr]

    47          2          0          0          0          0          0          0          0 [pm]

    48          2          0          0          0          0          0          0          0 [sync_supers]

    49          2          0          0          0          0          0          0          0 [bdi-default]

    50          2          0          0          0          0          0          0          0 [kintegrityd/0]

    51          2          0          0          0          0          0          0          0 [kintegrityd/1]

    52          2          0          0          0          0          0          0          0 [kintegrityd/2]

    53          2          0          0          0          0          0          0          0 [kintegrityd/3]

    54          2          0          0          0          0          0          0          0 [kintegrityd/4]

    55          2          0          0          0          0          0          0          0 [kintegrityd/5]

    56          2          0          0          0          0          0          0          0 [kintegrityd/6]

    57          2          0          0          0          0          0          0          0 [kintegrityd/7]

    58          2          0          0          0          0          0          0          0 [kblockd/0]

    59          2          0          0          0          0          0          0          0 [kblockd/1]

    60          2          0          0          0          0          0          0          0 [kblockd/2]

    61          2          0          0          0          0          0          0          0 [kblockd/3]

    62          2          0          0          0          0          0          0          0 [kblockd/4]

    63          2          0          0          0          0          0          0          0 [kblockd/5]

    64          2          0          0          0          0          0          0          0 [kblockd/6]

    65          2          0          0          0          0          0          0          0 [kblockd/7]

    66          2          0          0          0          0          0          0          0 [kacpid]

    67          2          0          0          0          0          0          0          0 [kacpi_notify]

    68          2          0          0          0          0          0          0          0 [kacpi_hotplug]

    69          2          0          0          0          0          0          0          0 [ata_aux]

    70          2          0          0          0          0          0          0          0 [ata_sff/0]

    71          2          0          0          0          0          0          0          0 [ata_sff/1]

    72          2          0          0          0          0          0          0          0 [ata_sff/2]

    73          2          0          0          0          0          0          0          0 [ata_sff/3]

    74          2          0          0          0          0          0          0          0 [ata_sff/4]

    75          2          0          0          0          0          0          0          0 [ata_sff/5]

    76          2          0          0          0          0          0          0          0 [ata_sff/6]

    77          2          0          0          0          0          0          0          0 [ata_sff/7]

    78          2          0          0          0          0          0          0          0 [ksuspend_usbd]

    79          2          0          0          0          0          0          0          0 [khubd]

    80          2          0          0          0          0          0          0          0 [kseriod]

    81          2          0          0          0          0          0          0          0 [md/0]

    82          2          0          0          0          0          0          0          0 [md/1]

    83          2          0          0          0          0          0          0          0 [md/2]

    84          2          0          0          0          0          0          0          0 [md/3]

    85          2          0          0          0          0          0          0          0 [md/4]

    86          2          0          0          0          0          0          0          0 [md/5]

    87          2          0          0          0          0          0          0          0 [md/6]

    88          2          0          0          0          0          0          0          0 [md/7]

    89          2          0          0          0          0          0          0          0 [md_misc/0]

    90          2          0          0          0          0          0          0          0 [md_misc/1]

    91          2          0          0          0          0          0          0          0 [md_misc/2]

    92          2          0          0          0          0          0          0          0 [md_misc/3]

    93          2          0          0          0          0          0          0          0 [md_misc/4]

    94          2          0          0          0          0          0          0          0 [md_misc/5]

    95          2          0          0          0          0          0          0          0 [md_misc/6]

    96          2          0          0          0          0          0          0          0 [md_misc/7]

    97          2          0          0          0          0          0          0          0 [linkwatch]

    98          2          0          0          0          0          0          0          0 [khungtaskd]

    99          2          0          0          0          0          0          0          0 [kswapd0]

   100          2          0          0          0          0          0          0          0 [ksmd]

   101          2          0          0          0          0          0          0          0 [aio/0]

   102          2          0          0          0          0          0          0          0 [aio/1]

   103          2          0          0          0          0          0          0          0 [aio/2]

   104          2          0          0          0          0          0          0          0 [aio/3]

   105          2          0          0          0          0          0          0          0 [aio/4]

   106          2          0          0          0          0          0          0          0 [aio/5]

   107          2          0          0          0          0          0          0          0 [aio/6]

   108          2          0          0          0          0          0          0          0 [aio/7]

   109          2          0          0          0          0          0          0          0 [crypto/0]

   110          2          0          0          0          0          0          0          0 [crypto/1]

   111          2          0          0          0          0          0          0          0 [crypto/2]

   112          2          0          0          0          0          0          0          0 [crypto/3]

   113          2          0          0          0          0          0          0          0 [crypto/4]

   114          2          0          0          0          0          0          0          0 [crypto/5]

   115          2          0          0          0          0          0          0          0 [crypto/6]

   116          2          0          0          0          0          0          0          0 [crypto/7]

   121          2          0          0          0          0          0          0          0 [kthrotld/0]

   122          2          0          0          0          0          0          0          0 [kthrotld/1]

   123          2          0          0          0          0          0          0          0 [kthrotld/2]

   124          2          0          0          0          0          0          0          0 [kthrotld/3]

   125          2          0          0          0          0          0          0          0 [kthrotld/4]

   126          2          0          0          0          0          0          0          0 [kthrotld/5]

   127          2          0          0          0          0          0          0          0 [kthrotld/6]

   128          2          0          0          0          0          0          0          0 [kthrotld/7]

   129          2          0          0          0          0          0          0          0 [pciehpd]

   131          2          0          0          0          0          0          0          0 [kpsmoused]

   132          2          0          0          0          0          0          0          0 [usbhid_resumer]

   162          2          0          0          0          0          0          0          0 [kstriped]

   243          2          0          0          0          0          0          0          0 [scsi_eh_0]

   246          2          0          0          0          0          0          0          0 [scsi_eh_1]

   253          2          0          0          0          0          0          0          0 [scsi_eh_2]

   254          2          0          0          0          0          0          0          0 [vmw_pvscsi_wq_2]

   438          2          0          0          0          0          0          0          0 [kdmflush]

   440          2          0          0          0          0          0          0          0 [kdmflush]

   457          2          0          0          0          0          0          0          0 [jbd2/dm-0-8]

   458          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

   550          1          0       2768        313        103         31          0        206 /sbin/udevd -d

   601          1       1000       6607        399        227        145          0        362 sapstart pf=/hana/shared/HED/profile/HED_HDB10_uyclsxxx.mdp.local

   610        601       1000     138893      71932      69195        185          0      25408 /usr/sap/HED/HDB10/uyclsxxx.mdp.local/trace/hdb.sapHED_HDB10 -d -nw -f /usr/sap/HED/HDB10/uyclsxxx.mdp.local/daemon.ini pf=/usr/sap/HED/SYS/profile/HED_HDB10_uyclsxxx.mdp.local

   628        610       1000    2688513    1944232     163035        544          0    1950291 hdbnameserver

   966        610       1000     837833     105659      44006         67          0     239339 hdbcompileserver

   968        610       1000     748149      94295      43165        753          0     145695 hdbpreprocessor

   989          2          0          0          0          0          0          0          0 [vmmemctl]

   995        610       1000   11782807   11072435    2853317        504          0    8294139 hdbindexserver -port 31043

  1038          2          0          0          0          0          0          0          0 [kdmflush]

  1080          2          0          0          0          0          0          0          0 [jbd2/sda1-8]

  1081          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

  1082          2          0          0          0          0          0          0          0 [jbd2/dm-4-8]

  1083          2          0          0          0          0          0          0          0 [ext4-dio-unwrit]

  1086          2          0          0          0          0          0          0          0 [xfs_mru_cache]

  1087          2          0          0          0          0          0          0          0 [xfslogd/0]

  1088          2          0          0          0          0          0          0          0 [xfslogd/1]

  1089          2          0          0          0          0          0          0          0 [xfslogd/2]

  1090          2          0          0          0          0          0          0          0 [xfslogd/3]

  1091          2          0          0          0          0          0          0          0 [xfslogd/4]

  1092          2          0          0          0          0          0          0          0 [xfslogd/5]

  1093          2          0          0          0          0          0          0          0 [xfslogd/6]

  1094          2          0          0          0          0          0          0          0 [xfslogd/7]

  1095          2          0          0          0          0          0          0          0 [xfsdatad/0]

  1096          2          0          0          0          0          0          0          0 [xfsdatad/1]

  1097          2          0          0          0          0          0          0          0 [xfsdatad/2]

  1098          2          0          0          0          0          0          0          0 [xfsdatad/3]

  1099          2          0          0          0          0          0          0          0 [xfsdatad/4]

  1100          2          0          0          0          0          0          0          0 [xfsdatad/5]

  1101          2          0          0          0          0          0          0          0 [xfsdatad/6]

  1102          2          0          0          0          0          0          0          0 [xfsdatad/7]

  1103          2          0          0          0          0          0          0          0 [xfsconvertd/0]

  1104          2          0          0          0          0          0          0          0 [xfsconvertd/1]

  1105          2          0          0          0          0          0          0          0 [xfsconvertd/2]

  1106          2          0          0          0          0          0          0          0 [xfsconvertd/3]

  1107          2          0          0          0          0          0          0          0 [xfsconvertd/4]

  1108          2          0          0          0          0          0          0          0 [xfsconvertd/5]

  1109          2          0          0          0          0          0          0          0 [xfsconvertd/6]

  1110          2          0          0          0          0          0          0          0 [xfsconvertd/7]

  1111          2          0          0          0          0          0          0          0 [xfsalloc/0]

  1112          2          0          0          0          0          0          0          0 [xfsalloc/1]

  1113          2          0          0          0          0          0          0          0 [xfsalloc/2]

  1114          2          0          0          0          0          0          0          0 [xfsalloc/3]

  1115          2          0          0          0          0          0          0          0 [xfsalloc/4]

  1116          2          0          0          0          0          0          0          0 [xfsalloc/5]

  1117          2          0          0          0          0          0          0          0 [xfsalloc/6]

  1118          2          0          0          0          0          0          0          0 [xfsalloc/7]

  1119          2          0          0          0          0          0          0          0 [xfseofblocks/0]

  1120          2          0          0          0          0          0          0          0 [xfseofblocks/1]

  1121          2          0          0          0          0          0          0          0 [xfseofblocks/2]

  1122          2          0          0          0          0          0          0          0 [xfseofblocks/3]

  1123          2          0          0          0          0          0          0          0 [xfseofblocks/4]

  1124          2          0          0          0          0          0          0          0 [xfseofblocks/5]

  1125          2          0          0          0          0          0          0          0 [xfseofblocks/6]

  1126          2          0          0          0          0          0          0          0 [xfseofblocks/7]

  1175          2          0          0          0          0          0          0          0 [kauditd]

  1272        610       1000     760274     106146      44191        630          0     155220 hdbwebdispatcher

  1378          1          0      23294        222        154         41          0      19022 auditd

  1394          1          0      62319        733        277         89          0      57003 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5

  1423          1          0       4596        197        118         10          0        108 irqbalance --pid=/var/run/irqbalance.pid

  1437          1          0       4744        227        165         13          0         76 rpcbind

  1455          1          0       5837        341        229         17          0        131 rpc.statd

  1481          1          0       7912        289        208         81          0       2634 dbus-daemon --system

  1511          1          0       1020        165        137          6          0         67 /usr/sbin/acpid

  1520          1          0       9502        945        702         83          0       2799 hald

  1521       1520          0       5100        301        245          4          0         75 hald-runner

  1553       1521          0       5630        275        235          6          0         75 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0

  1573       1521          0       4502        258        222          3          0         74 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket

  1799          1          0      16663        309        131        136          0        155 /usr/sbin/sshd

  1822          1          0       7684        529        375        173          0        198 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g

  1898          1          0      20324        861        635         36          0        150 /usr/libexec/postfix/master

  1907       1898          0      20387        862        640         72          0        177 qmgr -l -t fifo -u

  1908          1          0      44629        472        352          3          0        117 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages

  1919          1          0      44313       1523        458          1          0       1038 /usr/bin/python /usr/sbin/tuned -d -c /etc/tuned.conf

  1929          1          0      29323        345        191         14          0        355 crond

  1940          1          0       5385        123         77          5          0        120 /usr/sbin/atd

  1953          1          0      27085        164        129          3          0         72 /usr/bin/rhsmcertd

  1967          1          0       1016        137        120          3          0         67 /sbin/mingetty /dev/tty1

  1969          1          0       1016        137        120          3          0         67 /sbin/mingetty /dev/tty2

  1971          1          0       1016        137        120          3          0         67 /sbin/mingetty /dev/tty3

  1973          1          0       1016        137        120          3          0         67 /sbin/mingetty /dev/tty4

  1975          1          0       1016        138        120          3          0         67 /sbin/mingetty /dev/tty5

  1977          1          0       1016        138        120          3          0         67 /sbin/mingetty /dev/tty6

  2268          2          0          0          0          0          0          0          0 [flush-253:0]

  2406          2          0          0          0          0          0          0          0 [kdmflush]

  2417          2          0          0          0          0          0          0          0 [kdmflush]

  2715          2          0          0          0          0          0          0          0 [kdmflush]

  3017          2          0          0          0          0          0          0          0 [kdmflush]

  3085          2          0          0          0          0          0          0          0 [kdmflush]

  3560          1          0      30414        712        157          1          0        561 /usr/bin/python /usr/libexec/tuned/pmqos-static.py cpu_dma_latency=1

  3728          2          0          0          0          0          0          0          0 [xfsbufd/dm-7]

  3729          2          0          0          0          0          0          0          0 [xfssyncd/dm-7]

  3730          2          0          0          0          0          0          0          0 [xfsaild/dm-7]

  3731          2          0          0          0          0          0          0          0 [xfsbufd/dm-3]

  3732          2          0          0          0          0          0          0          0 [xfssyncd/dm-3]

  3733          2          0          0          0          0          0          0          0 [xfsaild/dm-3]

  3734          2          0          0          0          0          0          0          0 [xfsbufd/dm-5]

  3735          2          0          0          0          0          0          0          0 [xfssyncd/dm-5]

  3736          2          0          0          0          0          0          0          0 [xfsaild/dm-5]

  3737          2          0          0          0          0          0          0          0 [xfsbufd/dm-6]

  3738          2          0          0          0          0          0          0          0 [xfssyncd/dm-6]

  3739          2          0          0          0          0          0          0          0 [xfsaild/dm-6]

  3740          2          0          0          0          0          0          0          0 [xfsbufd/dm-2]

  3741          2          0          0          0          0          0          0          0 [xfssyncd/dm-2]

  3742          2          0          0          0          0          0          0          0 [xfsaild/dm-2]

  7008        610       1000   21582525   20783381    2938249        504          0   18011937 hdbindexserver -port 31040

  8965       1799          0      25099       1023        777        136          0        200 sshd: root@notty 

  8969       8965          0      14424        570        417         14          0        143 /usr/libexec/openssh/sftp-server

  9040       1799          0      25101       1109        852        136          0        202 sshd: root@pts/1 

  9044       9040          0      27085        471        365        212          0        106 -bash

  9062          2          0          0          0          0          0          0          0 [flush-253:3]

11057          1       1000     267694      15715       1669       2885          0     251696 /usr/sap/HED/HDB10/exe/sapstartsrv pf=/hana/shared/HED/profile/HED_HDB10_uyclsxxx.mdp.local -D -u hedadm

13614       9044          0      43868        667        512         28          0        178 sudo su - hedadm

13615      13614          0      36359        413        315          8          0        137 su - hedadm

13616      13615       1000      27114        478        369        212          0        135 -sh

23758        550          0       2767        279         70         31          0        205 /sbin/udevd -d

23759        550          0       3163        730        135         31          0        601 /sbin/udevd -d

23870          1          0      47395       1117        903        148          0       2864 /usr/sbin/vmtoolsd

24227       1898          0      20344        849        630         55          0        151 pickup -l -t fifo -u

25806          2          0          0          0          0          0          0          0 [flush-253:5]

26052          2          0          0          0          0          0          0          0 [flush-253:6]

29389          1          0      27512       2226        449        934          0      18777 /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile

29392          1        498     250193      16673      11687       3060          0     224935 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D

29422          1          0       6913       1178        552        743          0       1017 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile

[OK]

--

[OS_MEMORY]  Operating system information about memory: (2016-04-25 15:08:19 172 Local)

/proc/meminfo:

MemTotal:       154912776 kB

MemFree:         2799548 kB

Buffers:          154964 kB

Cached:         37542020 kB

SwapCached:            0 kB

Active:         119601352 kB

Inactive:       30299076 kB

Active(anon):   112269892 kB

Inactive(anon): 22791284 kB

Active(file):    7331460 kB

Inactive(file):  7507792 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       6291448 kB

SwapFree:        6291448 kB

Dirty:              1784 kB

Writeback:             0 kB

AnonPages:      112203468 kB

Mapped:         23260764 kB

Shmem:          22857644 kB

Slab:             738012 kB

SReclaimable:     614844 kB

SUnreclaim:       123168 kB

KernelStack:        5440 kB

PageTables:       389820 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:    83747836 kB

Committed_AS:   137382676 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      542572 kB

VmallocChunk:   34359179772 kB

HardwareCorrupted:     0 kB

AnonHugePages:         0 kB

HugePages_Total:       0

HugePag

/proc/vmstat:

nr_free_pages 699887

nr_inactive_anon 5697821

nr_active_anon 28067473

nr_inactive_file 1876948

nr_active_file 1832865

nr_unevictable 0

nr_mlock 0

nr_anon_pages 28050867

nr_mapped 5815191

nr_file_pages 9424246

nr_dirty 446

nr_writeback 0

nr_slab_reclaimable 153711

nr_slab_unreclaimable 30792

nr_page_table_pages 97455

nr_kernel_stack 680

nr_unstable 0

nr_bounce 0

nr_vmscan_write 433

nr_writeback_temp 0

nr_isolated_anon 0

nr_isolated_file 0

nr_shmem 5714411

numa_hit 2115178222

numa_miss 0

numa_foreign 0

numa_interleave 114388

numa_local 2115178222

numa_other 0

nr_anon_transparent_hugepages 0

pgpgin 1545840710

pgpgout 2793009976

pswpin 433

pswpout 433

pgalloc_dma 2

pgalloc_dma32 155436

pgalloc_normal 2120506030

pgalloc_movable 0

pgfree 2121363732

pgactivate 9295851

pgdeactivate 46976

pgfault 2170754705

pgmajfault 1254

pgrefill_dma 0

pgrefill_dma32 46976

pgrefill_normal 0

pgrefill_movable 0

pgsteal_dma 0

pgsteal_dma32 433

pgsteal_normal 2869225

pgsteal_movable 0

pgscan_kswapd_dma 0

pgscan_kswapd_dma32 2720

pgscan

/proc/sysvipc/shm:

       key      shmid perms                  size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime                   rss                  swap

         0   51118080   740                  1024   601   610      1  1000    79  1000    79 1460640107 1460640107 1460640106                     0                     0

     11001   51150849   600             268435456   610  7074      7  1000    79  1000    79 1460643197 1460643197 1460640107             268435456                     0

1446068590   51183618   660                   117   628  7008      6  1000   500  1000    79 1460643180 1460640195 1460640109                  4096                     0

351291274   51216387   660               1060911   628  7008     24  1000   500  1000    79 1460643180 1460640195 1460640109               1060864                     0

785145360   51249156   660                   108   628  7008      6  1000   500  1000    79 1460643181 1460640195 1460640109                  4096                     0

/proc/sys/kernel:

/proc/sys/vm/overcommit_memory:

0

/proc/sys/vm/panic_on_oom:

0

/proc/sys/vm/oom_kill_allocating_task:

0

/proc/sys/vm/compact_memory:

Could not open!

/proc/sys/vm/extfrag_threshold:

500

/proc/sys/vm/oom_dump_tasks:

1

/proc/sys/vm/would_have_oomkilled:

0

/proc/sys/vm/overcommit_ratio:

50

/proc/sys/vm/page-cluster:

3

/proc/sys/vm/dirty_background_ratio:

10

/proc/sys/vm/dirty_background_bytes:

0

/proc/sys/vm/dirty_ratio:

40

/proc/sys/vm/dirty_bytes:

0

/proc/sys/vm/dirty_writeback_centisecs:

500

/proc/sys/vm/dirty_expire_centisecs:

3000

/proc/sys/vm/nr_pdflush_threads:

0

/proc/sys/vm/swappiness:

60

/proc/sys/vm/nr_hugepages:

0

/proc/sys/vm/nr_hugepages_mempolicy:

0

/proc/sys/vm/hugetlb_shm_group:

0

/proc/sys/vm/hugepages_treat_as_movable:

0

/proc/sys/vm/nr_overcommit_hugepages:

0

/proc/sys/vm/lowmem_reserve_ratio:

256 256 32

/proc/sys/vm/drop_caches:

0

/proc/sys/vm/min_free_kbytes:

49818

/proc/sys/vm/extra_free_kbytes:

0

/proc/sys/vm/unmap_area_factor:

0

/proc/sys/vm/percpu_pagelist_fraction:

0

/proc/sys/vm/max_map_count:

15800000

/proc/sys/vm/laptop_mode:

0

/proc/sys/vm/block_dump:

0

/proc/sys/vm/vfs_cache_pressure:

100

/proc/sys/vm/legacy_va_layout:

0

/proc/sys/vm/zone_reclaim_mode:

0

/proc/sys/vm/min_unmapped_ratio:

1

/proc/sys/vm/min_slab_ratio:

5

/proc/sys/vm/stat_interval:

1

/proc/sys/vm/mmap_min_addr:

4096

/proc/sys/vm/numa_zonelist_order:

default

/proc/sys/vm/scan_unevictable_pages:

0

/proc/sys/vm/memory_failure_early_kill:

1

/proc/sys/vm/memory_failure_recovery:

1

[OK]

--

[THREADS]  Running threads: (2016-04-25 15:08:19 172 Local)

2324[thr=1062]: JobWrk0003, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2582[thr=1321]: JobWrk0004, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2748[thr=1454]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

54292[thr=13394]: JobWrk0005, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

522995[thr=28979]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

879757[thr=16378]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2216776[thr=23586]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

5[thr=0]: <SYSTEM>, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eae65113 in __libc_poll+0x53 (libc.so.6)

691[thr=980]: TimerThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

720[thr=983]: TraceSegmentCompressor, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: 983, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "Search for trace files to compress", detail: "", command: "" at 0x00007f08ee660c29 in Execution::ContextFunctions::printProgramCounter(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&)+0x19 at ContextFunctions.cpp:887 (libhdbbasis.so)

2118[thr=979]: MemoryCompactor, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eb127aa1 in sem_timedwait+0x61 (libpthread.so.0)

2123[thr=981]: NetworkChannelCompletionThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eae6ef33 in __epoll_wait_nocancel+0x2a (libc.so.6)

2124[thr=982]: NetworkChannelCompletionThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eae6ef33 in __epoll_wait_nocancel+0x2a (libc.so.6)

2140[thr=984]: WakeupThread, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eb1287dd in __read_nocancel+0x24 (libpthread.so.0)

2236[thr=985]: SignalQueue, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "SignalQueue", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2259[thr=988]: PreprocessorUtil, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "PreprocessorUtil", method: "", detail: "", command: "" at 0x00007f08eae32a7d in __nanosleep_nocancel+0x24 (libc.so.6)

2261[thr=986]: PreprocessorPool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: 0, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "PreprocessorPool", method: "", detail: "", command: "" at 0x00007f08eae65113 in __libc_poll+0x53 (libc.so.6)

2264[thr=987]: SelfWatchDog, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "SelfWatchDog", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2275[thr=1032]: Pool, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Pool", method: "waiting for work", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2316[thr=1058]: JobexWatchdog, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobexWatchdog", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2318[thr=1059]: JobexMainDispatcher, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "Generic", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2320[thr=1060]: JobWrkDiag, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobexDiag", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

2322[thr=1061]: JobWrk0002, TID: -1, UTID: -1, CID: -1, LCID: 0, parent: ---, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: ---, StmtCtx: ---, type: "JobWorker", method: "", detail: "", command: "" at 0x00007f08eae6b199 in syscall+0x19 (libc.so.6)

[OK]

--

[MEMORY_OOM]  Information about current out of memory situation: (2016-04-25 15:08:19 175 Local)

OUT OF MEMORY occurred.

Host: uyclsxxx.mdp.local

Executable: hdbpreprocessor

PID: 968

Failed to allocate 32816 byte.

Current callstack:

1: 0x00007f08ee86c88c in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, unsigned long, bool)+0xb8 at PoolAllocator.cpp:777 (libhdbbasis.so)

2: 0x00007f08ee86d5c0 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x250 at PoolAllocator.cpp:1449 (libhdbbasis.so)

3: 0x00007f08eae2e651 in __alloc_dir+0x60 (libc.so.6)

4: 0x00007f08ee9382b4 in System::UX::opendir(char const*)+0x20 at SystemCallsUNIX.cpp:75 (libhdbbasis.so)

5: 0x00007f08ee745de8 in FileAccess::DirectoryEntry::findFirst()+0x14 at SimpleFile.cpp:600 (libhdbbasis.so)

6: 0x00007f08ee74bd9b in FileAccess::DirectoryEntry::DirectoryEntry(char const*)+0x467 at SimpleFile.cpp:99 (libhdbbasis.so)

7: 0x00007f08ee622ec0 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x250 at TraceSegment.cpp:158 (libhdbbasis.so)

8: 0x00007f08ee6e9f29 in Execution::Thread::staticMainImp(void**)+0x7e5 at Thread.cpp:467 (libhdbbasis.so)

9: 0x00007f08ee6eadfd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

Memory consumption information:

[No information available.]

Out of memory reasons:

!:! <1> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <2> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <3> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

At most one OOM dump is written in approximately 86400s.

Raw memory information (1):

Allocated=1305010176 (accounted=1305010176, unaccounted=0)

Reserved size for libraries (data segments only)=758431744 (data segments size (libraries)=762748928)

Raw Pool Allocator size=446193664 (including huge alignment raw size=0, V=0)

Allocate Only size=71471104(limited=16777216, unlimited=54693888 [including emergency allocators], unreserved Allocate Only size=0)Raw Limited Allocate Only Allocator size=16777216

Stack size=26042368

Free list size=0

Memory failure free list size=0

Big StackReservationScope reservations=0 (limited=0, unlimited=0)

VirtualAlloc/MMap = 2228224

JumpTable = 643072

StatAllocOwnBlocks =524288

OOMReservationAlloc (not part of 'allocated'!) = 0

(System memory: 10253490249728 - 10252943147008 = 547102720, + data sagements=1305534464)

Delta for size of data segments: dataSegmentSize=762748928, reservedSizeForDataSegments=758431744

Compaction tasks

Last compaction: 220. Reason: IPMM (other process). Request ID=172823.

ReserveRequest=13.82mb (14491648b), ShrinkRequest=4194303.99tb (4611686018427383808b)

Compaction tasks for compaction 220:

  Limits:  TempLimit=988.55mb (1036574720b), ProcessLimit=138.79gb (149034151936b).

  Alloc status: Used=217.32mb (227880229b), Allocated=988.55mb (1036574720b).

  Fragmentation: 79%.

  Freelist: Size=0b.

* Forced garbage collection (fragmentation too big).

  Alloc status: Used=203.50mb (213388581b), Allocated=1002.37mb (1051066368b).

  Fragmentation: 80%.

  Freelist: Size=0b.

  Compactor status before compaction: Size=0b, Freeable=0b.

* Weak garbage collection.

* Forced garbage collection..

  Alloc status: Used=203.50mb (213388581b), Allocated=1002.37mb (1051066368b).

  Fragmentation: 80%.

  Freelist: Size=0b.

Result: Reserved=0b.

Called with parameter 'forced garbage collection'.

Called with parameter 'forced' (wait for lock).

Inter Process Memory Management

IPMM shared memory segment: SAP_IPMM_0028_HED

SId=        Process secondary identifier

AB=         Allocated bytes by allocators for heap and stacks (inclusive allocator cache)

UA=         Unaccounted memory (by unlimited allocators for exceptions etc.), maybe overcommit

U=          Used bytes for heap and stacks (without allocator cache)

FSL=        Free segment list size

SMAS/SMDS=  System memory traffic information (accumulation of allocations/deallocations)

PAL=        Process allocation limit

TPAL=       Temp process allocation limit

Blocked=    Memory (TPAL!) blocked/reserved for this process

W=          Weight of process

[0] PID=628 (state=S), SId=300674672, compactors active, alive, process name: hdbnameserver

    AB=8.18gb (8788430848b), UA=0b, U=5.85gb (6283392859b), FSL=0b, SMAS=7.69tb (8460664975360b), SMDS=7.68tb (8452727377920b), PAL=138.79gb (149034151936b), TPAL=8.43gb (9056288768b), W=100

[1] PID=966 (state=S), SId=300674673, compactors active, alive, process name: hdbcompileserve

    AB=1.56gb (1680842752b), UA=0b, U=300.78mb (315401201b), FSL=0b, SMAS=18.19gb (19538411520b), SMDS=17.32gb (18607349760b), PAL=138.79gb (149034151936b), TPAL=2.20gb (2363437056b), Blocked=2.20gb (2363805696b), W=100

[2] PID=968 (state=R), SId=300674674, compactors active, alive, process name: hdbpreprocessor

    AB=1.21gb (1305010176b), UA=0b, U=217.32mb (227881461b), FSL=0b, SMAS=9.32tb (10253490249728b), SMDS=9.32tb (10252943147008b), PAL=138.79gb (149034151936b), TPAL=1.45gb (1564049408b), W=100

[3] PID=995 (state=S), SId=300674675, compactors active, alive, process name: hdbindexserver

    AB=32.38gb (34772930560b), UA=0b, U=28.70gb (30824124599b), FSL=0b, SMAS=6.48tb (7130062503936b), SMDS=6.45tb (7096140505088b), PAL=138.79gb (149034151936b), TPAL=32.38gb (34772930560b), W=100

[4] PID=1272 (state=S), SId=300674676, compactors active, alive, process name: hdbwebdispatche

    AB=1.26gb (1358028800b), UA=0b, U=501.49mb (525850774b), FSL=0b, SMAS=12.05tb (13258559606784b), SMDS=12.05tb (13257979600896b), PAL=138.79gb (149034151936b), TPAL=1.26gb (1358028800b), W=100

    1 thread is waiting for IPMM lock

[5] PID=7008 (state=S), SId=300674677, compactors active, alive, process name: hdbindexserver

    AB=69.45gb (74576596992b), UA=0b, U=62.54gb (67162769381b), FSL=0b, SMAS=9.22tb (10138308907008b), SMDS=9.15tb (10064582815744b), PAL=138.79gb (149034151936b), TPAL=69.45gb (74576596992b), W=100

    1 thread is waiting for IPMM lock

GLOBAL_MAX_ALLOCATION_LIMIT=138.79gb (149034151936b)

SHARED_MEMORY=21.69gb (23298207744b)

OOM_RESERVATION_ALLOCATORS=288.54mb (302563328b)

CODE_SIZE=1.62gb (1742049280b)

Sum TPAL (including shm, code size and IPMM reservation and IPMM OOM reservation)=138.79gb (149034151936b) (cached sum=138.79gb (149034151936b)), IPMM reservation=0b, emergency reservation=128mb (134217728b)

#processes=6, weight sum=600, cached weight sum=600, #checks for cached allocation limit sum=9549644, #terminated processes since last reset=0

Early exits are currently allowed. Dumping of IPMM flight recorder is currently enabled.

15 early exits since creation of IPMM, reclaim-memory-counter=1527688. Reclaim memory 1527655 failed, only 253943808b were free.

Last reclaim memory started to wait for IPMM lock since 2016-04-25 15:08:18.872.

Last reclaim memory got IPMM lock at 2016-04-25 15:08:19.141.

IPMM flight recorder is not verbose.

User ids respected in shm calculation: [1000]

Parallel memory reclaim tasks are activated.

This is an opt build.

Peak used memory: 125.63gb (134897761776b) with corresponding values (Heap=102.31gb (109857504752b), Shared=21.69gb (23298207744b), Code=1.62gb (1742049280b)).

Peak heap memory: 102.31gb (109857504752b). Peak shared memory: 21.69gb (23298207744b). Peak code size: 1.62gb (1742049280b)