Hello,

yesterday we wanted to install AFL plugin to our HANA database. The installation was succesfull, but afterwards our HANA database doesn't want to start. We can't find anything helpfull in the log files (attached all traces).

HANA DB Version: 1.00.85.01.399276

After issuing HDB start command, it takes around 45 minutes and then throwing Timeout error.

We tryied to restart the whole server, but it didn't help.

Could you please help to solve this issue?

Bellow is the indexserver log:

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089018 i Basis            TraceStream.cpp(00396) : ==== Starting hdbindexserver, version 1.00.85.01.399276 (NewDB100_REL), build linuxx86_64 not set 2015-01-14 17:43:42 ld7272.wdf.sap.corp gcc (SAP release 20140516, based on SUSE gcc47-4.7.2_20130108-0.15.45) 4.7.2 20130108 [gcc-4_7-branch revision 195014]

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089267 i Basis            TraceStream.cpp(00401) : MaxOpenFiles: 1048576

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089301 i Memory           MallocProxy.cpp(01181) : Installed malloc hooks

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089304 i Basis            ProcessorInfo.cpp(00746) : Using GDT segment limit to determine current CPU ID

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089305 i Basis            Timer.cpp(00629) : Using RDTSC for HR timer

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089305 i Memory           AllocatorImpl.cpp(01219) : Allocators activated

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089306 i Memory           AllocatorImpl.cpp(01235) : Using big block segment size 67108864

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089308 w Environment      Environment.cpp(00286) : Changing environment set IMSLERRPATH=/usr/sap/HAN/HDB00/exe//

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.089309 w Environment      Environment.cpp(00286) : Changing environment set IMSLSERRPATH=/usr/sap/HAN/HDB00/exe//

[37346]{-1}[-1/-1] 2016-05-06 09:15:04.096804 w Environment      Environment.cpp(00286) : Changing environment set SSL_WITH_OPENSSL=0

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.762076 i assign           TREXIndexServer.cpp(00623) : assign to volume 2 started

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.762833 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HAN/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.764371 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HAN/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.765992 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/HANA/log/mnt00001/hdb00002/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.767501 i FileIO           FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/HANA/data/mnt00001/hdb00002/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.769510 i HistoryManager   HistoryManager.cpp(00492) : Maximum garbage collection parallelity set to 144

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.769691 i PhysicalPageAcce DataVolumeImpl.cpp(00543) : Open volume file(s) in "/HANA/data/mnt00001/hdb00002/" (readOnly= 0, diagMode= 0)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.769711 i PhysicalPageAcce DataVolumeImpl.cpp(00556) : --confMaxFileSize= 0MB, fsysMaxFilesSize= 2048000MB --> 2048000MB

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.769888 i PhysicalPageAcce DataVolumeImpl.cpp(00601) : --opened volume file "datavolume_0000.dat"

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781315 i Crypto           RootKeyStore.cpp(00157) : Empty SSFS cache: reading from SSFS (ID=PERSISTENCE/ROOTKEY)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781381 i Crypto           RootKeyStore.cpp(00104) : RSecSSFs: Entering function "RSecSSFsGetRecord" [/HDB/IMP/NewDB100_PATCH_REL/src/sys/src/spine/src/krn/rsec/rsecssfs.c 850]

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781392 i Crypto           RootKeyStore.cpp(00104) : RSecSSFs: Using explicitly set configuration data [/HDB/IMP/NewDB100_PATCH_REL/src/sys/src/spine/src/krn/rsec/rsecssfs.c 3907]

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781423 i Crypto           RootKeyStore.cpp(00104) : RSecSSFs: Data file "/usr/sap/HAN/SYS/global/hdb/security/ssfs//SSFS_HAN.DAT" opened for read [/HDB/IMP/NewDB100_PATCH_REL/src/sys/src/spine/src/krn/rsec/rsecssfs.c 2351]

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781439 i Crypto           RootKeyStore.cpp(00104) : RSecSSFs: Key file "/usr/sap/HAN/SYS/global/hdb/security/ssfs/SSFS_HAN.KEY" not found, using default key [/HDB/IMP/NewDB100_PATCH_REL/src/sys/src/spine/src/krn/rsec/rsecssfs.c 1271]

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.781876 i Crypto           RootKeyStore.cpp(00104) : RSecSSFs: Exiting function "RSecSSFsGetRecord" with return code 0 (message: <No message available>) [/HDB/IMP/NewDB100_PATCH_REL/src/sys/src/spine/src/krn/rsec/rsecssfs.c 911]

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782145 i PersistenceManag PersistenceManagerImpl.cpp(04014) : Restart page version 1 loaded (96bytes)

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782173 i PersistenceManag PersistenceManagerImpl.cpp(04039) : Initial maximum known TID after restart: 460382606

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782175 i Logger           PersistenceManagerImpl.cpp(04047) : Newest known master commit position: 0xd8c56f082

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782179 i Logger           PersistenceManagerImpl.cpp(04057) : Known last prepare commit position on volume 1: 0x22126e88

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782192 i Logger           PersistenceManagerImpl.cpp(04057) : Known last prepare commit position on volume 3: 0x132c855ac3

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782193 i Logger           PersistenceManagerImpl.cpp(04057) : Known last prepare commit position on volume 4: 0xc6d706d83

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782194 i Logger           PersistenceManagerImpl.cpp(04057) : Known last prepare commit position on volume 5: 0x71c883

[37615]{-1}[-1/-1] 2016-05-06 09:15:04.782200 i PersistenceManag PersistenceManagerImpl.cpp(04111) : Known DTX volume set [1,3,4,5,1024], 0 RTT entries

[37615]{-1}[-1/-1] 2016-05-06 09:15:12.236865 i PersistenceManag PersistenceSessionRegistry.cpp(00254) : Start loading open sessions and history cleanup files

[37615]{-1}[-1/-1] 2016-05-06 09:15:12.593347 i Logger           PersistenceSessionRegistry.cpp(01062) : Open session count at restart: 1/1, max known TID at restart: 460382606

[37615]{-1}[-1/-1] 2016-05-06 09:15:12.593367 i PersistenceManag PersistenceSessionRegistry.cpp(00261) : Loading 1 open session(s) and 1 history cleanup file(s) finished in 0.712954 seconds;

[37615]{-1}[-1/-1] 2016-05-06 09:15:12.864095 i Service_Startup  ContMgr.cc(00078) : Initializing system catalog.

[37615]{-1}[-1/-1] 2016-05-06 09:15:13.064796 i Service_Startup  ContMgr.cc(00200) : Initializing system catalog done.

[37731]{-1}[-1/-1] 2016-05-06 09:15:14.117828 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01092) : LoadMultiplePageBlocksAtStartup

[37731]{-1}[-1/-1] 2016-05-06 09:15:14.117887 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01110) : allocate 347 segments requested to load and collect information about superblocks to read...

[37731]{-1}[-1/-1] 2016-05-06 09:15:18.616832 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01260) : collecting information done in 4498msec.

[37731]{-1}[-1/-1] 2016-05-06 09:15:18.617153 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01291) : SuperblockPrefetchCalculation:

  allocationLimit=482491495213

  inUseSize=3883578777

  prefetchSize=382886333148

  prefetchCount=5705

[37731]{-1}[-1/-1] 2016-05-06 09:15:18.617172 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01684) : set SuperblockPrefetchCount=5705 (max: 32767)

[37731]{-1}[-1/-1] 2016-05-06 09:15:18.617174 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01297) : trigger load of superblocks...

[37628]{-1}[-1/-1] 2016-05-06 09:15:18.744715 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01601) : load process: 0% (1 of 356) : overall 501MB/s.

[37731]{-1}[-1/-1] 2016-05-06 09:15:37.533549 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01342) : trigger load of superblocks done after 18916msec.

[37628]{-1}[-1/-1] 2016-05-06 09:15:37.562763 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01620) : i/o finished: read 356 superblock(s) within 18945ms = 1202MB/s overall.

[37731]{-1}[-1/-1] 2016-05-06 09:15:37.562841 i RowStorePageAcce AbsolutePageAccessImpl.cpp(01350) : finished loading all 347 segments by reading 356 superblocks after 23444msec.

[37731]{-1}[-1/-1] 2016-05-06 09:15:37.622566 i Service_Startup  CheckpointMgr.cc(00703) : 347 RS segments loaded from Persistency in 23.5 sec (945.02127659574467MB/s)

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.627666 i Service_Startup  MemSystem.cc(00143) : total memory size (MB) = 516796

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.627807 i Logger           PersistenceManagerImpl.cpp(02273) : Preparing log recovery handler, start position 0xd8c56f0c6

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.627980 i Logger           LoggerImpl.cpp(00444) : Setting log segment backup timeout to 86400s, original value was 0s

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.663639 w PersistenceManag PersistenceManagerImpl.cpp(03513) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.663662 i Logger           PersistenceManagerImpl.cpp(03708) : Deactivating REDO logging for log replay

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.663664 i Logger           PersistenceManagerImpl.cpp(03719) : Start termination of rollback(s) open in restart/backup savepoint

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.665658 i Logger           PersistenceManagerImpl.cpp(03727) : Termination of rollback(s) open in restart/backup savepoint finished in 0.0001 seconds;

[37615]{-1}[-1/-1] 2016-05-06 09:15:37.665678 i Logger           PersistenceManagerImpl.cpp(03732) : Start garbage collection of history files

[37551]{-1}[-1/-1] 2016-05-06 09:15:47.665775 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:15:57.665912 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:07.666016 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:17.666149 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:27.666243 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:37.666344 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:47.666436 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:16:57.666549 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:17:07.666645 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:17:17.666755 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:17:27.666908 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37551]{-1}[-1/-1] 2016-05-06 09:17:37.667020 i PersistenceManag PersistenceManagerImpl.cpp(01282) : Garbage collection of history files running: 0 cleanup files of 1 done

[37615]{-1}[-1/-1] 2016-05-06 09:17:47.177448 i Logger           PersistenceManagerImpl.cpp(03743) : Garbage collection of history files finished: 1 cleanup files in 129.511 seconds;

[37615]{-1}[-1/-1] 2016-05-06 09:17:47.177487 i Logger           PersistenceManagerImpl.cpp(03759) : Expecting savepoint 222299 log record as first log record at 0xd8c56f0c6

[37615]{-1}[-1/-1] 2016-05-06 09:17:47.177685 i Logger           RecoveryHandlerImpl.cpp(00936) : RecoveryQueueCount used: 64

[37622]{-1}[-1/-1] 2016-05-06 09:17:47.927329 i Logger           RecoveryHandlerImpl.cpp(00374) : Stopping parsing of open segment LogSegment[0:0xddef47ac0/Writing][GUID=3620CC37-146B-20160505-220549-1FE0000001/PrevGUID=3620CC37-146B-20160505-212730-3920000001/PersGUID=3620CC37-146B-20160413-091157-09D0000001/RestoreGUID=54A09A46-442E-20131025-055130-0120000001] at file position 4096, unused rest 1073737728B, termination reason: Invalid buffer header found at position 0x1000 in segment 0xddef47ac0

[37615]{-1}[-1/-1] 2016-05-06 09:17:47.955063 i Logger           PersistenceManagerImpl.cpp(03827) : Starting log replay at position 0xd8c56f0c6

[37615]{-1}[-1/-1] 2016-05-06 09:17:47.955073 i Logger           RecoveryHandlerImpl.cpp(01472) : Triggering recovery of remaining log

[38874]{-1}[-1/-1] 2016-05-06 09:17:49.268008 i Logger           PersistenceManagerImpl.cpp(03041) : Found savepoint 222299 log record

[37551]{-1}[-1/-1] 2016-05-06 09:17:57.955542 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58230265920 and time: <none> (2%)

[37551]{-1}[-1/-1] 2016-05-06 09:20:57.957031 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58263811904 and time: 2016-04-15 17:28:56.452446+01:00 (5%)

[37551]{-1}[-1/-1] 2016-05-06 09:21:47.957542 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58297362944 and time: <none> (7%)

[37551]{-1}[-1/-1] 2016-05-06 09:26:37.960083 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58330904576 and time: 2016-04-15 17:29:30.283030+01:00 (10%)

[37551]{-1}[-1/-1] 2016-05-06 09:26:47.960205 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58364445504 and time: 2016-04-15 17:29:46.116009+01:00 (12%)

[37551]{-1}[-1/-1] 2016-05-06 09:31:57.963045 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58397984960 and time: 2016-04-15 17:30:03.709249+01:00 (15%)

[37551]{-1}[-1/-1] 2016-05-06 09:32:27.963377 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58448281600 and time: <none> (18%)

[37551]{-1}[-1/-1] 2016-05-06 09:32:57.963636 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58532135168 and time: <none> (24%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:07.963730 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58666277824 and time: <none> (34%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:17.963822 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58733368512 and time: <none> (39%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:27.963953 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58783692672 and time: 2016-04-15 17:35:56.967064+01:00 (42%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:38.820403 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58834013056 and time: 2016-04-15 17:36:54.529099+01:00 (46%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:48.820447 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 58884339072 and time: 2016-04-15 17:38:22.929161+01:00 (50%)

[37551]{-1}[-1/-1] 2016-05-06 09:33:58.820550 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59001750912 and time: 2016-04-15 17:43:12.286839+01:00 (58%)

[37551]{-1}[-1/-1] 2016-05-06 09:34:48.821137 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59035285504 and time: <none> (61%)

[37551]{-1}[-1/-1] 2016-05-06 09:36:38.822096 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59204025024 and time: 2016-04-15 17:45:20.215160+01:00 (73%)

[37551]{-1}[-1/-1] 2016-05-06 09:36:48.822185 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59303638528 and time: 2016-04-15 17:57:56.870844+01:00 (80%)

[37551]{-1}[-1/-1] 2016-05-06 09:36:58.822274 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59392411072 and time: 2016-04-15 18:10:00.324347+01:00 (86%)

[37551]{-1}[-1/-1] 2016-05-06 09:37:08.844271 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59456303040 and time: 2016-04-15 18:29:00.734755+01:00 (91%)

[37551]{-1}[-1/-1] 2016-05-06 09:37:18.844362 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59499546496 and time: 2016-04-19 00:56:47.289513+01:00 (94%)

[37551]{-1}[-1/-1] 2016-05-06 09:37:28.844455 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59531929728 and time: 2016-04-26 04:56:24.324397+01:00 (96%)

[37551]{-1}[-1/-1] 2016-05-06 09:37:38.844559 i Logger           RecoveryHandlerImpl.cpp(00663) : Redo done up to position: 59568842880 and time: 2016-05-04 11:09:52.077684+01:00 (99%)

[38874]{-1}[-1/-1] 2016-05-06 09:37:40.604635 i Logger           RecoveryHandlerImpl.cpp(01866) : Log recovery cannot expect any more data at position 0xddef47b00, load states:

  - LogPartition[/HANA/log/mnt00001/hdb00002/:0]: #segments 103/104 processed, 104/104 loaded, 104 total, last loaded LogSegment[0:0xddef47ac0/Writing][GUID=3620CC37-146B-20160505-220549-1FE0000001/PrevGUID=3620CC37-146B-20160505-212730-3920000001/PersGUID=3620CC37-146B-20160413-091157-09D0000001/RestoreGUID=54A09A46-442E-20131025-055130-0120000001]

[38874]{-1}[-1/-1] 2016-05-06 09:37:40.604696 i Logger           RecoveryHandlerImpl.cpp(02166) : Recovery finished at log position 0xddef47b00

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.604713 i Logger           RecoveryHandlerImpl.cpp(01520) : Finishing log recovery, waiting for in-process and in-load segments

[38850]{-1}[-1/-1] 2016-05-06 09:37:40.625582 i Logger           RecoveryHandlerImpl.cpp(03341) : Signaling finish wait barrier

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.627890 i Logger           LogSegment.cpp(00593) : Closing open log segments after log recovery completed at position 0xddef47b00

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.628042 i Logger           LogSegment.cpp(00676) : Closing open on-disk segment LogSegment[0/103:0xddef47ac0/GUID=3620CC37-146B-20160505-220549-1FE0000001/PrevGUID=3620CC37-146B-20160505-212730-3920000001,TS=2016-05-05 23:05:49.510644/Writing/0x0]@0x00007ff034124440 after log recovery completed at position 0xddef47b00

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.629115 i Logger           LoggerImpl.cpp(00933) : Replayed 88707600000B (84598MB) of log in 1193.45 seconds; 70.8852MB/s; max known TID=462402851

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.629222 i Logger           LoggerImpl.cpp(00723) : Starting logger with 8 log buffers per partition, 1024KB each, segment size 1024MB, log mode normal

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.646217 i Logger           RecoveryHandlerImpl.cpp(01767) : Termination of 0 indoubt transactions was executed in 0.008134 seconds;

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.648902 i Logger           PersistenceManagerImpl.cpp(03940) : Activating REDO logging after log replay

[37615]{-1}[-1/-1] 2016-05-06 09:37:40.686650 i LogReplay        RowStoreTransactionCallback.cc(00222) : starting master-slave DTX consistency check

[37615]{-1}[-1/-1] 2016-05-06 09:37:41.690414 i PersistenceManag PersistenceManagerImpl.cpp(04842) : DTX slave 5 reports rollback at restart completion; pre-restart RTT still awaits restart of volumes [1,3,4,1024]

[37615]{-1}[-1/-1] 2016-05-06 09:37:41.690432 i PersistenceManag PersistenceManagerImpl.cpp(04842) : DTX slave 3 reports rollback at restart completion; pre-restart RTT still awaits restart of volumes [1,4,1024]

[37615]{-1}[-1/-1] 2016-05-06 09:37:41.690437 i PersistenceManag PersistenceManagerImpl.cpp(04842) : DTX slave 4 reports rollback at restart completion; pre-restart RTT still awaits restart of volumes [1,1024]

[37615]{-1}[-1/-1] 2016-05-06 09:37:41.690457 i LogReplay        RowStoreTransactionCallback.cc(00310) : finished master-slave DTX consistency check

[37615]{-1}[-1/-1] 2016-05-06 09:37:42.732183 i ContainerDirecto ContainerNameDirectoryImpl.cpp(01124) : Load ContainerNameDirectory...

[38066]{-1}[-1/-1] 2016-05-06 09:37:42.732254 i PersistenceManag VirtualFileStatsProxy.cpp(00130) : Initialize VirtualFile statistics...

[37874]{-1}[-1/-1] 2016-05-06 09:37:42.732254 i PersistenceManag VirtualFileStatsProxy.cpp(00186) : Initialize VirtualFile LOB statistics...

[37615]{-1}[-1/-1] 2016-05-06 09:37:46.847969 i ContainerDirecto ContainerNameDirectoryImpl.cpp(01185) : ContainerNameDirectory loaded 978484/978484 in 4.12 sec

[37874]{-1}[-1/-1] 2016-05-06 09:37:49.319309 i PersistenceManag VirtualFileStatsProxy.cpp(00220) : VirtualFile LOB statistics initialized: 3507009 CD entries in 6.59 sec

[38066]{-1}[-1/-1] 2016-05-06 09:37:49.974404 i PersistenceManag VirtualFileStatsProxy.cpp(00165) : VirtualFile statistics initialized: 978484/978484 CD entries in 7.24 sec

Hi Arpit,

Options suggested by Saravanan and Davide should work

If you prefer using the command line statements(as in Oracle) please refer the link below for the RECOVER STATEMENT commands under the section "Recovery to a point in time"

Note: the commands will change in case of a tenant Database as per the scenarios mentioned in the link

https://help.sap.com/saphelp_hanaplatform/helpdata/en/c4/205f73bb571014ba46d209981bc1f5/content.htm

Hope it helps

Sunil

Dear Moho,

the following is shown in the log file.

20131025-055130-0120000001] at file position 4096, unused rest 1073737728B, termination reason: Invalid buffer header found at position 0x1000 in segment 0xddef47ac0

May something is corrupt. Please check the sap notes for a solution.

Best regards

Martin

Hi Moho,

Looking at the logs it does not explain the cause of the Database startup issue

The logs here are only between 09:15 AM to 09:37 AM, do you have more entries in the log after that? As you mentioned the process timed out after 45 mins above?

As it is a SPS08 system I believe this a Single container system, is it a distributed system? Can you share more details on the Architecture

Also during the AFL upgrade did the HANA Database have a planned stop and start or did it undergo a crash at some point?

More information would help and I would strongly suggest you to open a case with SAP Support meanwhile to make sure you have enough support

Sunil

Hello,

the upgrade of AFL were without problem - but the system doesn't want to start anymore.

It's a single container system.

Not distributed.

After issuing HDB start the console is waiting for HANA to start - as the timeout is set to 45 minutes it's giving error: Timeout.

The provided log is everything I have. Nothing more is written to the file.

For name server all I have is:

[11172]{-1}[-1/-1] 2016-05-06 11:04:49.955059 w Backup           BackupMonitor_TransferQueue.cpp(00048) : Master index server not available!

[11172]{-1}[-1/-1] 2016-05-06 11:04:49.955131 w Backup           BackupMonitor_TransferQueue.cpp(00048) : Master index server not available!

[11172]{-1}[-1/-1] 2016-05-06 11:04:49.955180 w Backup           BackupMonitor_TransferQueue.cpp(00048) : Master index server not available!

[11172]{-1}[-1/-1] 2016-05-06 11:04:49.955181 w Backup           BackupMonitor_TransferQueue.cpp(00332) : Indexserver not answering.

Best regards

I am having a problem where the SIGNATURE.SMF file is not created, despite using SAPCAR correctly.  Have you ever seen this before?

Hi WeiFeng,

Why are you going for landscape to check memory overall consumption ? Thats going to give you a by host/service drilldown of memory allocated / used.

60% on what exactly? What are you comparing this to and why? What columns are you using for comparison?

Show the result of your query here and then clarify your questions. Clarify your HANA revision as well.

Thanks,

Lucas de Oliveira

Hi guys

Lately, we are experiencing several Out-of-Memory situations on our SAP Landscape, after it was migrated to SAP HANA Database

It could occur randomly in ERP, NW PO or BW, since all of them were migrated from SQL Server to HANA Database

Basically, SAP Instance freezes, and only option is to restart it.

Under alert it shows

preprocessor on host uyclsxxx port 31002 was restarted at 2000-01-01 00:00:00.0000000.

webdispatcher on host uyclsxxx port 31006 was restarted at 2000-01-01 00:00:00.0000000.

Memory Overview seems to be normal and there are no peaks thay might hint Memory is below required

I was able to obtain some diagnostic files and all seems to be related to HANA Server unable to allocate more memory.

Nameserver, Indexserver and webdispatcher Out-Of-Memory dumps all shows the same information.

Any idea on how to troubleshoot these out-ot-memory errors? Bear in mind it could occur at any time, even on non-working hours or weekend when there are no activity on the SAP System

[MEMORY_OOM]  Information about current out of memory situation: (2016-05-04 21:38:05 562 Local)

OUT OF MEMORY occurred.

Host: uycls157.mdp.local

Executable: hdbnameserver

PID: 22961

Failed to allocate 32816 byte.

Out of memory reasons:

!:! <1> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <2> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <3> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <4> MemoryPool::allocate(big/huge) failure.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <5> BigBlockAllocator failure, no precharge.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <6> IPMM early exit.

    /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

At most one OOM dump is written in approximately 86400s.

[OS_MEMORY]  Operating system information about memory: (2016-05-04 21:38:26 465 Local)

/proc/meminfo:

MemTotal:       154912776 kB

MemFree:         1644660 kB

Buffers:          155040 kB

Cached:         37670000 kB

SwapCached:            0 kB

Active:         120563596 kB

Inactive:       30454160 kB

Active(anon):   113194280 kB

Inactive(anon): 22856080 kB

Active(file):    7369316 kB

Inactive(file):  7598080 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       6291448 kB

SwapFree:        6291448 kB

Dirty:              8836 kB

Writeback:             0 kB

AnonPages:      113192916 kB

Mapped:         23249932 kB

Shmem:          22857644 kB

Slab:             749560 kB

SReclaimable:     624456 kB

SUnreclaim:       125104 kB

KernelStack:        5416 kB

PageTables:       417168 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:    83747836 kB

Committed_AS:   136213928 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      542572 kB

VmallocChunk:   34359179772 kB

HardwareCorrupted:     0 kB

AnonHugePages:         0 kB

HugePages_Total:       0

HugePag

/proc/vmstat:

nr_free_pages 411165

nr_inactive_anon 5714020

nr_active_anon 28298570

nr_inactive_file 1899520

nr_active_file 1842329

nr_unevictable 0

nr_mlock 0

nr_anon_pages 28298229

nr_mapped 5812483

nr_file_pages 9456260

nr_dirty 2209

nr_writeback 0

nr_slab_reclaimable 156114

nr_slab_unreclaimable 31276

nr_page_table_pages 104292

nr_kernel_stack 677

nr_unstable 0

nr_bounce 0

nr_vmscan_write 433

nr_writeback_temp 0

nr_isolated_anon 0

nr_isolated_file 0

nr_shmem 5714411

numa_hit 3058504450

numa_miss 0

numa_foreign 0

numa_interleave 114388

numa_local 3058504450

numa_other 0

nr_anon_transparent_hugepages 0

pgpgin 2381614114

pgpgout 3146033750

pswpin 433

pswpout 433

pgalloc_dma 2

pgalloc_dma32 155436

pgalloc_normal 3065907965

pgalloc_movable 0

pgfree 3066476428

pgactivate 9823911

pgdeactivate 46976

pgfault 3140334036

pgmajfault 1262

pgrefill_dma 0

pgrefill_dma32 46976

pgrefill_normal 0

pgrefill_movable 0

pgsteal_dma 0

pgsteal_dma32 433

pgsteal_normal 2869225

pgsteal_movable 0

pgscan_kswapd_dma 0

pgscan_kswapd_dma32 2720

pgsc

/proc/sysvipc/shm:

       key      shmid   perms                  size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime                        rss                     swap

         0   67010560   740                  1024 22935 22943      1  1000    79  1000    79 1461791037 1461791037 1461791037                     0                 0

     11001   67043329   600             268435456 22943 24417      7  1000    79  1000    79 1461792167 1461792167 1461791037             268435456    0

1446068590   67076098   660                   117 22961 24369      6  1000   500  1000    79 1461792151 1461791363 1461791039                  4096        0

351291229   67108867   660               1060911 22961 24369     24  1000   500  1000    79 1461792151 1461791363 1461791039               1060864   0

785145360   67141636   660                   108 22961 24369      6  1000   500  1000    79 1461792151

1461791363 1461791040                  4096        0

Hello

I have SalesData coming from VBAK. I have SalesDocument Date which I joined with M_TIME_DIMENSION - DATE_SAP.

I can get the CALWEEK, CALMONTH, YEAR now. But below are my questions.

1. How to get "Lastweek" from any Date?

2.Create a report with Last Week Sales.

3. Compare This Year Last week with Last Year Last week.

The report should dynamically run. ( a User Prompt or a Input Variable would also be helpful)

I looked into below links but not sure it answers my questions.

https://help.sap.com/saphelp_hanaplatform/helpdata/en/20/e23edc7519101482eae3271722de36/content.htm?frameset=/en/20/f5bb79751910148e7198206fbd199f/frameset.htm&current_toc=/en/2e/1ef8b4f4554739959886e55d4c127b/plain.htm&node_id=111

Implementation of WTD, MTD, YTD in HANA using Input Parameters Derived from Table

Implementation of WTD, MTD, YTD in HANA using Script based Calc View calling Graphical Calc view

Re: Filter by date variable

Get Date Dimension – Quarter start date, Quarter end date, Week Start Date, Month Start Date, Month End Date, Week End Date, Current Quarter start date, Start date of previous quarter and End date of previous quarter

Hi Martin,

Troubleshooting a OOM issue based on the information we have at hand is not straight forward

Looking at your current version of SPS10 102.01 and the known issues we are aware so far your case could be matching the symptoms indicated in the below SAP note

You will need to follow the instructions from the note to generate a stack and check the statement_memory_limit setting on the HANA system to be able to confirm the issue matches the scenario from the note

http://service.sap.com/sap/support/notes/2289105

The note suggests a work around and a solution to upgrade to the latest Maintenance release on SPS10

Please validate the issue matches this note in your case before taking the next steps

I would also suggest you to open a case with SAP Support at the same time to make sure you have their inputs on this

Hope it helps

Sunil

Dear Colleagues,

I have a query on BW on HANA licencing. If we have BW on HANA exist in our landscape, Can we do HANA modelling using HANA Studio in parallel to normal BW modelling in RSA1?? Do we need any licensing required for this?? Please suggest.

Best Regards,

Krishna

Krishna,

for BW modeling, yes you can use HANA Studio for that. you will need to do some plug in installs,

see here: https://scn.sap.com/thread/3686050

Hi There,

I have been looking into a solution for this issue I have been experiencing over and over after several different upgrade path tries. When I am trying to upgrade hana from SSP9 Rev 92 to SPS11 Rev 112, it finishes the upgrade with a warning but successfully.

Note: 'SAP HANA Database' update finished with warning, because of failed deployment of HANA content (part of 'SAP HANA Database' archive). For next steps, check SAP Note 1795885.

The server status is green but I couldn't find why it is saying it finished with a warning and a delivery unit failed. It is one of the core delivery unit installations too based on the log file it points to which is a bit scary if the core stuff fails. From the log file it says;

19:59:06.115 - INFO:    Output line 49:  All server processes started on host '....' (worker).

19:59:06.116 - INFO:    Output line 50: Importing delivery units...

19:59:06.238 - INFO:    Output line 51: Importing delivery unit HCO_INA_SERVICE

19:59:08.113 - INFO:    Output line 52: Importing delivery unit HANA_DT_BASE

19:59:23.768 - INFO:    Output line 53: Importing delivery unit HANA_IDE_CORE

20:00:04.038 - INFO:    Output line 54: Importing delivery unit HANA_TA_CONFIG

20:00:15.238 - INFO:    Output line 55: Importing delivery unit HANA_UI_INTEGRATION_SVC

20:00:36.530 - INFO:    Output line 56: Importing delivery unit HANA_UI_INTEGRATION_CONTENT

20:00:38.658 - INFO:    Output line 57: Importing delivery unit HANA_XS_BASE

20:01:00.778 - INFO:    Output line 58: Importing delivery unit HANA_XS_DBUTILS

20:01:03.301 - INFO:    Output line 59: Importing delivery unit HANA_XS_LM

20:01:57.841 - INFO:    Output line 60: Importing delivery unit HDC_ADMIN

20:02:09.398 - INFO:    Output line 61: Importing delivery unit HDC_IDE_CORE

20:04:17.729 - INFO:    Output line 62: Importing delivery unit HDC_SEC_CP

20:04:20.088 - INFO:    Output line 63: Importing delivery unit HDC_XS_BASE

20:04:22.286 - INFO:    Output line 64: Importing delivery unit HDC_XS_LM

20:04:23.895 - INFO:    Output line 65: Importing delivery unit SAPUI5_1

20:06:55.450 - INFO:    Output line 66: Importing delivery unit SAP_WATT

20:08:36.472 - INFO:    Output line 67: Importing delivery unit HANA_BACKUP

20:08:48.267 - INFO:    Output line 68: Importing delivery unit HANA_HDBLCM

20:08:58.724 - INFO:    Output line 69: Importing delivery unit HANA_SEC_BASE

20:09:46.282 - INFO:    Output line 70: Importing delivery unit HANA_SEC_CP

20:10:36.519 - INFO:    Output line 71: Importing delivery unit HANA_ADMIN

20:17:51.898 - INFO:    Output line 72:

20:17:51.899 - INFO:    Output line 73:  ##############################################################

20:17:51.899 - INFO:    Output line 74:  # Problems occurred while importing delivery units (content).

20:17:51.899 - INFO:    Output line 75:  # To complete the content update, refer to SAP Note 1795885.

20:17:51.899 - INFO:    Output line 76:  #############################################################

20:17:51.899 - INFO:    Output line 77:

20:17:53.531 - INFO:    Output line 78: Log file written to '/var/tmp/hdb_AR2_hdblcm_update_2016-05-06_19.48.05/hdbupd.log' on host '....'.

20:17:53.963 - INFO:    Program terminated with exit code 2

20:17:53.964 - INFO: -----------------------------------------------------------

20:17:53.964 - INFO: END: Updating SAP HANA Database (start: 19:49:43.034 duration: 00:28:10.929)

20:17:53.964 - INFO: -----------------------------------------------------------

SO, did it fail installing HANA_ADMIN delivery unit? How can I tell which delivery unit or units failed and how can I retrieve those failed ones and import those? Do I use import feature with the system user via Hana Studio to re-import? How can I extract those delivery units from the installation?

I am thinking about upgrading to SPS10 rev 100 first then to sps11 112 next, not sure if that would help?

Thanks in advance for your help.

It is mentioned to me that restoring an older revision of HANA backup on newer revision of HANA is okay as far as the Backup and Recovery guide goes. In our case the restore is being attempted from rev 93 to 112. I have not seen this statement myself but it was quoted and seems rather odd why this dangerous statement would be part of the official guide.

Knowing that HANA has different schemas and settings between two revisions, how would it be suggested to restore on higher version, I don't quite understand? I can clearly see some base schemas and certain privileges get wiped out if you do such restore after my clean install of 92 and restoring a backup of on clean install of 112. What are your thoughts?

Is there a safe way to do such restore while copying all data, schemas and additional user/roles/pther catalog objects implemented on top of the base installation to the higher version of HANA? The main objective here is to replicate whatever was configured and populated already on one instance, to another one without having to upgrade the source instance to prevent disruption to the source use and simply switch over to the new one once the testing is completed with the new version.

PS: The other alternative of course I am trying is the install a fresh copy of 93 on another instance, restore the backup on it then upgrade that to 112 which has its own issues too which I am checking on another thread.

Thanks for your input.

I have now also tried upgrading from sps 9 rev 93 to sps 10 rev 100, I get the same warning and error within the log stating the delivery units are not installed. It seems to be a common issue. I checked some SAP notes about missing delivery units issue but nothing points to this specific issue in the consecutive patches as far as I have seen.

Is there a way to know which delivery unit is missing and how I can get it from the installation? Sorry it may be an easy question, not used the delivery units patching for the base packages before however i worked with the delivery units in terms of importing and activating.

Hi Sarada,

If you know the start date of the the date-field then you can get

last week: ADD_DAYS(date-field, -7)

last year: ADD_YEARS(date-field, -1)

Regards

Raj

Digging into it a bit more, I found this;

07:43:33.453 - ERR : Cannot import delivery unit /hana/shared/AR3/global/hdb/auto_content/HANA_XS_LM.tgz

08:05:36.361 - INFO: Messages from trace files

08:05:36.323 - INFO:   indexserver_alert_llbpal60.trc: More than 50 relevant messages found, check file.

08:05:36.325 - INFO:   preprocessor_alert_llbpal60.trc: [36471]{-1}[-1/-1] 2016-05-07 08:03:42.368419 e preprocessor     PreprocessorImpl.cpp(01280) : inconsistent result: relativeLength < 0 , docKey=_SYS_REPO:ACTIVE_OBJECTen.CDATA:158032, relativeLength=-1, htmlOffset=521, tokenLength=8

Checking the log files, only delivery unit failed to install from the base hana installation is the HANA_XS_LM. I saw this in the log files;

08:26:46.331 - INFO:       Output line 46: ACTIVATION ERROR: object: "", package: "", timestamp: 2016-05-07 08:26:46.2720000

08:26:46.331 - INFO:       Output line 47: : code 129: Transaction was marked in invalid

08:26:46.331 - INFO:       Program terminated with exit code 1

08:26:46.332 - ERR :     Error importing delivery units: regi returned exit code '1'

I have found the DU under the /hana/shared folder and re-imported using the Hana Studio. It installed and activated fine on its own. Am I supposed to re-import and re-activate other things after or do you think that is sufficient? When I selected it from the Studio to import during the wizard, it already said all components were installed for this delivery unit and active. Makes me wonder if there is an error thrown for no reason?

Hello Mert,

Indeed, in the HANA Administration Guide, the following is stated:

"A SAP HANA database cannot be recovered to an SAP HANA database with a lower software version. The SAP HANA software version used for the recovery must always be the same version or higher than the SAP HANA database used to create the data backup or storage snapshot."

It's not that dangerous a statement imho, since the backup you're restoring will consist of all data in the database (all deployed HANA content, all data of a SAP installation in the SAPABAP1 (default) schema, database users and their schema's, roles, etc.).

The potential problem you sugest is related to the database content, which you are completely replacing with the content from the backup. You will have the revision 93 database content running on HANA with 112 binaries. I don't see a problem with that right away. You might get issues when you subsequently want to update this 112 to another revision, as then you'll get the post processing for the new revision on your restored data. I have no field experience with this last scenatrio, but have performed a restore to a higer revision without problems.

Kindest regards, Wilbert

Hi Nishith,

Did you have to add the field WCB_COCO_NUM to MV45AFZZ in order to make the functionality work?

I am struggling to get the condition DCHB determined in sales order. SAP is not able to determine the condition contract number.

Regards,
Abhijit